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Abstract 

Singular Spectrum Analysis and many other subspace-based methods 
of signal processing are implicitly relying on the assumption of close prox- 
imity of unperturbed and perturbed signal subspaces extracted by the 
Singular Value Decomposition of special "signal" and "perturbed signal" 
matrices. In this paper, the analysis of the largest principal angle between 
these subspaces is performed in terms of the perturbation expansions of 
the corresponding orthogonal projectors. Applicable upper bounds are 
derived. The main attention is paid to the asymptotic case when the 
length of the time series tends to infinity. Results concerning conditions 
for convergence, the rate of convergence, and the main terms of proximity 
are presented. 

1 Introduction 

Though the first publication on the subspace-based methods of signal process- 
ing traditionally runs back to [IJ, the explosive development of these methods 
ocurred during the last 30 - 35 years. In numerous scientific and engineering 
areas from climatology [2J and meteorology ^ to acoustics j4], from petroleum 
geology [51 pp 41 - 51] to marine sciences [B], from human |7] and animal [S] 
population dynamics to noise reduction problems \W , many researches indepen- 
dently proposed similar "signal-subspace" ideas and applied them to time series 
of their own interest. 

Omitting details that are specific and important for each version of the 
method, the general scheme of the signal-subspace approach can be explained as 
follows. Consider a time series Ftv = {xq, . . . , xn-i) treated as a "signal". (Note 
that Fjv can be one-dimensional or multidimensional, real-valued or complex- 
valued.) This series is linearly transformed into a certain L x K "signal matrix" 
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H. (The entries of H can also be either real- valued or complex- valued.) It is 
supposed that the signal Fat and the transformation Fat > H are such that 
d rankH < min(L, K) and the linear space spanned by columns of matrix H 
contains important information about the series Fjv- The latter space is called 
signal subspace. 

dcf 

In terms of the "covariance" matrix A = HH* this means that the 
eigenspace Uq corresponding to zero eigenvalue of the matrix A is not degener- 
ate and that we are interested in the orthogonal complement Vq of Uq. Note 
that the matrix H must be a certain structured matrix. Depending on the data 
at hand and on the problem under study, Hankel, Toeplitz, block-Hankel, and 
other similar matrices are used in practice. 

Assume now that the perturbed series FAr((5) = FAr + (5Ejv is observed rather 
than the unperturbed series Fjv, where Ejv — (cq, . . . , ejv-i) is a "noise series" 
and S stands for a formal perturbation parameter. Thus, instead of the "signal 
matrix" H we observe the perturbed matrix H((5) = H -I- SE, where the "noise 
matrix" E is constructed from the series Eat in the same manner as H is built 
from the series F^r. Note that generally there is no a priori assumption on the 
structure of the noise series. For example, the origin of E^r can be random or 
deterministic. 

Consider the Singular Value Decomposition (briefly, SVD) of H(5). If 5 is 
small, then continuity considerations show that the linear space Uf}^((5) spanned 
by d leading left singular vectors of this SVD can serve as an approximation of 
U^. Thus we can expect that useful (though approximate) information about 
the signal Fn can be extracted from Uj]"((5) with the help of a certain special 
technique. Since the linear space Uj-((5) is spanned by d leading eigenvectors of 

the matrix A{6) H((5)H*((5), this procedure is usually formulated in terms 
of the eigendecomposition of the perturbed "covariance" matrix A(^). 

These or similar ideas under different names (for example. Eigenvector Fil- 
tering |3] , Extended Empirical Orthogonal Functions [5] , Karhunen-Loeve Time 
Series Analysis [7], Singular System Analysis [TU], Dynamic Principal Compo- 
nent Analysis |11| ) or without any special names are used in many publications. 
The so-called Singular Spectrum Analysis (whether treated as in [12 or as in 
|13j ) has the same origin. 

The most well-known example of this scheme is the following. Suppose that 
a one-dimensional series F^r is governed by some minimal Linear Recurrent 
Formula (briefly, LRF) 



of order d. Transform F into the LxK Hankel "trajectory" matrix H with rows 
{xj, . . . , XK+j-i); where j — 0, . . . , L — 1 and L + K ^ N + 1. If min(i, K) > d, 
then rankH = d = dimUj}- and dimUo — K — d > 0. 

The knowledge of Vq (or, equivalently, of Uq) provides an essential in- 
formation about the series Fjy. In particular, if we consider Schubert basis 



d 



X. 




n> d 



k=l 
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y = {Yi, . . . , Yt-d) of the linear space Uq (see [T?] or [T31 ch. 5] for relations 
between this basis and LRFs), then the first element Yi of 3^ determines the 
minimal LRF governing the series F^r. Finding the roots of the related charac- 
teristic polynomial is the usual goal in practical investigations. 

There is a large number of publications dedicated to methods and algorithms 
for estimation of these roots by the use of the linear space UfJ-((5) in the case 
when E is a white-noise series (see, for example, chapters 11 and 13 in |15|). 
Methods named as MUSIC and ESPRIT are presently the most popular. Stan- 
dard references for the origins of these approaches are [16] and [17] . A modern 
description of ESPRIT together with the first-order perturbation analysis of its 
performance can be found in |18| . 

Let us discuss the aim, the technique, and the results of the paper. The 
main goal is to estimate the difference between Vq and Uj-((5) in the case of 
long signals. 

We describe both unperturbed Vq and perturbed U^((5) signal subspaces by 
their orthogonal projectors Pq and Pf}"((5), respectively. The spectral norm (also 
known as the operator norm) || • || is used to measure the difference between 
these projectors. (Spectral norm of a matrix can be defined as its maximal 
singular value.) Since the dimension of Vq and Uj]"((5) is the same, the norm 
||P^((5) — P^ll is the sine of the largest principal angle between the linear spaces 
U^((5) and U^. (For example, this follows from [THl ch. 3 §39].) 

To estimate this norm from above we use the perturbation expansions of 
Pq{S). These expansions (see formulas (2.2 1 and (2.4 1 of Section |2]) are valid 
under the condition \S\ < Sq for some positive ^nd can be straightforwardly 
deduced from the classical monograph |20| ch. II §3]. It follows from ( |2.4[ ) that 
the projector Pi]"(5) is continuous in S as \S\ < Sq. In other words, if \S\ < Sq 
then Uj]-(i5) can be continuously traced back to U^. 

Suitable upper bounds for ||Pj]^((5) — Pj-|| lead to the solution of several 
problems related to the proximity of perturbed and unperturbed signal sub- 
spaces in the case of long signals. Solving these problems, we restrict ourselves 
to real- valued signals governed by LRFs. 

More precisely, we consider an infinite signal series F — {xq, xi, . . . , a;„, . . .) 
which is governed by a minimal linear recurrent formula of order d. As it was 
already described, finite segments F^v = {xq,xi, . . . ,xn-i) of the series F are 
transformed into signal Lx K Hankel matrices H, where L depends on N. Thus 
the signal subspace Uq with the related projector P^ depends on N too. The 
noise series is supposed to be infinite as well and the expansions (2.2 1 and (2.4 1 
are valid for all S such that < = ^o(A^)- 

If N ^ oo, then we come to several problems related to the asymptotic 
behavior of ||P^((5) — Po"||- One problem is the behavior of So{N) as iV — >■ cxd 
for a certain signal and noise series. If 6o{N) — >■ oo, then we can use expansions 
of Pf{- (^ ) for any fixed 6 provided that N is sufficiently large. If i5o (N) is bounded 
from above (and is separated from zero) , then expansions are valid at a certain 
interval around zero. 

Another problem is related to the proximity of Pq (S) and Pq. In the present 
paper, we indicate three origins of the proximity P^{d) « P^f . The first origin 
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can be formulated only in terms of the extremal singular values of the matri- 
ces H and E. Roughly speaking, the difference ||Pjj-(5) — P;j-|| is small if the 
"signal-to-noise ratio" ||H||/||E|| is large and the "pseudo-condition number" of 
the matrix HH^, which is defined as the ratio of the maximal and minimal 
positive eigenvalues of this matrix, is not very large. If these two conditions 
hold asymptotically, then ||Po ((5) — Fq\\ 0. 

Nevertheless, the convergence ||Pj]'((5) — Pq\\ — > can take place even if the 
"signal-to-noise ratio" is bounded from above. This happens when both (column 
and row) pairs of linear spaces produced by matrices H and E are asymptotically 
orthogonal. 

The third origin is connected with the condition EEi^ /K — >■ I, where I stands 
for the identity matrix. This case corresponds to the white-noise "noise series". 
Note that if the "noise series" is random, then the perturbed operator is also 
random. Moreover, for fixed 5 and N the inequality \6\ < 5o{N) holds only with 
a certain probability. 

One more interesting problem related to the convergence ||Pj^(^) — Pj]-|| — )■ 
is the form of the main term of the approximation P J- (6) w P;j^ . Since S is fixed 
and N oo, this term does not necessarily have to be linear in 6. 

Let us explain the style of our results for these asymptotic proximity prob- 
lems. For this purpose we use simplified but typical examples. More general 
examples are considered in Section [T2| 



Example 1.1. Let the signal series be defined as a;„ — cos(27ra;n -I- 0) with 
to € (0, 1/2). Then the pseudo-condition number of the matrix HH"'' is bounded 
from above as N ^ oo. 

a) Suppose that the "noise series" is a linear stationary random series defined on 
a probability space (il, J^, P). (Exact definitions and restrictions can be found 



in Lemma 3.2 of Section 3.2 ) If L/N — > a G (0, 1) as iV — t' oo, then for a certain 



ri' e J" with V{n') = l, for any w e fi', and for any S € 



limsupv/^V(hijVp" ||P(J-(,5) -P(]-|| < 5i and 

N 

limsup7V(lniV)-i ||PfJ-(<5) - P^ - SY^^Ml < 62 

N 

where (^Vq^-* stands for the linear in 6 term of the difference Pi}"((5) — Pq. It 
should be mentioned that the "signal-to-noise ratio" ||H||/||E|| tends almost 
surely to infinity as A'^ — >■ 00 in this example. 

b) Assume that the "noise series" is also oscillating and defined by e„ = 
cos(27ra;'n + 0') with (0,1/2) 3 lj' ^ ut. If mm{L,K) — > 00, then for any 
6 € (-1/4, 1/4) and for N > Na{6) 

\\P^{S)-P^\\ = \5\0{l/mm{L,K)) and 
\\P^{S) -P^- L{5)\\ - S'0{l/mm{L^K')), 



where L((5) is the non-linear in S operator defined in Theorem 2.5 This case 



corresponds to the asymptotic biorthogonality of matrices H and E. 
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c) If white noise stands for the "noise series" and L ^ Lq = const > d, then for 
a certain 6a > 0, for some il' G T with P(r2') = 1, for any \6\ < Sq, and for any 

uj en' 



hmsup v/A^(lnlnAr)-i ||pJ-(^) 



N 

Um sup iV (In In A^) " 

N 



< c\ \b\ and 



where the non-Hnear in b operator T((5) is defined in Theorem 2.6 The proof 
of this case is based on the almost sure (briefly, a.s.) convergence EE"'"//^ I. 

The present paper is organized as follows. Section |2] contains basic theo- 
retical results. General definitions and notation, as well as formulas for the 
perturbation expansion of the perturbed projector Pj]^((5) are placed in Section 



In Section 



2.2 



general upper bounds for the difference 



|P(f('5)-P,fl 



are 



constructed and discussed. In Theorem 2.3 we derive the inequality (2.151 such 
that Pj]"(^) — Pfj" iff the right-hand side of (2.151 equals zero. (The conditions 



for the equality of the perturbed and unperturbed projectors are discussed in 



for the proximity of P^ ((5) and P 



Section 2.2.1 ) Corollaries 2.1 and 2.2 contain interpretable sufficient conditions 



The rest of Section |2.2| is devoted to the main ter ms o f the approxi mation 
Pfj-((5) fa Pjj-. These results are collected in Theorems 2.4 - 2.6 of Section 2.2.3 

In Section [3] we perform the asymptotic analysis of ||P^((5) — P^|| in the 
case when the size of L x if matrices H and E increases. More precisely, we 
consider conditions for the convergence ||Pj^((5) — Po"|| — >■ as max(L, K) — > 00 
and study the rate of this convergence. 

Two general assertions of this kind are presented in Section [3?T| Starting from 
Section |3.2| the case of Hankel matrices H and E is examined more thoroughly. 
Several more or less general time series of finite rank are taken as signal and 
noise series and a number of inequalities similar to those of Example |1.1| are 
derived. To obtain these inequalities we need to study the asymptotic behavior 
for the minimal and maximal positive singular values of the corresponding LxK 
Hankel matrices as L + K ^ 00. Results for deterministic series are collected 
in Lemma |3. II 

The case of stationary random "noise series" is of special interest. To estimate 
norms of the related Hankel matrices, see Lemma |3.2| we use recent results 
published in [2T 



and 



Proposition 7.1 of Appendix 2 contains proofs of 
In its turn, inequalities concerning 



inequalities we apply in our considerations 
white-noise "noise series" are based on results of Section [7T2| which is also placed 
in Appendix 2. 

In Section [4] we present several examples of "signal" and "noise" series such 
that both the exact rates of convergence ||PfJ-((5) — PqW and the related 
main terms are expressed in the explicit form. These examples are rather simple, 
but they show that refined versions of the inequalities of Section [XTJcan provide 
precise results. 



5 



In general, the term "signal-subspace method" implies that a method pro- 
vides a small error if the perturbed subspace Uq (S) is close to the unperturbed 
\Jq . Propositions related to the estimation of LRFs and to the so-called Least- 
Square ESPRIT confirm this statement. These propositions are placed in Sec- 
tion |5l 

Singular Spectrum Analysis (briefly. SSA), which can also be considered as 
a signal-subspace method, shows a different effect. Namely, the SSA reconstruc- 
tion may not converge to a signal even if ||Pg-(J) — Po"|| — as TV — oo. The 
related discussion can be found in Section 15.31 

Proofs of assertions are placed in Appendix 1. It its turn. Appendix 2 collects 
auxiliary results related to random "noise series". Note that some simple but 
laborsome asymptotic considerations are used without proofs. 

2 Perturbations of signal subspaces 

In what follows we do not distinguish linear operators from their matrices. In 
particular, we use the sign of transposition B"^ instead of the conjugation sign 
B*. To avoid misunderstanding, we sometimes use the notation Om for the zero 
vector in K*^ as well as the notation Omp for the zero operator M*^ H' M.^ . 

2.1 Perturbation series 

Let H be a real-valued non-zero i-^ matrix. Then A HH""" is a self- 
adjoint semi-definite operator A : M.^ i-> M^. We assume that d = rankH < 
min(L, K). Denote S the set of eigenvalues of A. Then G E C [0, oo). 

Consider one more real-valued non-zero matrix E : i-> and put 
H((5) = H 5E. Then 

A{6) =^ H((5)H((5)^ = A + ^A^^^ + (J^A^^) = A + B(^), 

where A(i) = HET-t-EH^^, A^^) = EE^, and B((5) = S A'-^'> + A^^'> . Note that 
both A(i) and A^^) are self-adjoint operators, and A(S) is positive semi-definite 
for any 5 € M. 

We interpret H as a "signal matrix" and E as a "noise matrix". It is supposed 
that we observe the perturbed matrix H{S) and are interested in the "signal 
subspace" spanned by the columns of the matrix H. 

The signal subspace is the orthogonal complement Vq to the eigenspace Ug, 
corresponding to the zero eigenvalue of the matrix HH'^. If Pq stands for the 
orthogonal projector onto Uq, then = I — Pq is the orthogonal projector 
onto Uq-. (Here I is the identical operator M.^.) 

Our goal is to compare the perturbed projector Pj]^((5) with the unperturbed 
projector Fq. In what follows, we set fiynin = min{/x e E such that n > 0}. 

The following assertion can be easily deduced from general results given in 
|20l ch. II §3]. Denote Sq the pseudoinverse to HH""". Set Sq"-* = — Pq and 
S^,''^ = for fc > 1. 
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Theorem 2.1. 1. Let (^o > and assume that 

||B(5)|| </imm/2 (2.1) 

for all S e {—So, 5o)- Then the perturbed projector Pq (5) admits the expansion 

oo 

P^{S) = Pi+J2Wp{S), (2.2) 



where 



and 



WpiS) = i-ir Wp(Zi,...,/p+i) (2.3) 

;i+...+ip+i=p, 'j>o 



Wp(Zi, . . . , Zp+i) = s^'^)b(5)s('^^ . . . s^''')b(5)sJ,'-+^ 

In addition, 

P^(5) = P^ + 5]5«V("\ (2.4) 

n=l 



where 



E (-1)' E V(")(s,^), (2.5) 



p=[n/2] si + ...+Sp=n, Si = l,2 

h+...+lp+i=p, lj>0 



(si, . . . , Sp), £ = (h, Ip+i), and 

V("^(s,^) - s[,'^U(^i)s[,'^^ . . . A^'^'^sl^''+'\ 



Remark 2.1. 1. Both series (2.2) and (2.4 1 converge in the spectral norm. 
2. Denote 

B{S) ^ \S\ IIA'i'll +(52||A(2)||. (2.6) 



If (5o > and B{So) = /iinin/2, then the inequality (2.11 is valid for all S such 
that \S\ < So- 

3. Since HH'^ — X]u>o/^-'^P' then So = X]/i>o ■'^m/M- (Here and further we 



write X]/j>o ii^stead of X]s3^>o-) '^^ t° show that ||Sq'°''|| = I/m'" 



min 



for 



4. The coefficient Vq^-* of the linear in S term of the right-hand side of (2.4 1 has 



any fc > 
4. The c( 
the form 

V^,'^ =PoA(i)So + SoA«Po 



(2.7) 

= PqEHTSo + SoHE^Pq. 



7 



5. Accurate calculations show that 



V|,') =PoA(2)So + SoA(2)Po 
+ PoA(i)PoA(i)S^ + PoAfi^S^A^i^Po 
+ SgA(i)PoA(i)Po - PoA(i)SoA(i)So 
- SoA(i)PoA(i)So - SoA(i)SoA(i)Po. 



(2.1 



6. The term Wi((5) in the right-hand side of ( |2.2[ ) can be expressed as 
Wi(5) = PoB(5)So-HSoB(5)Po 

= <5 V(^) + <52(PoA(2)So + SoA(2)Po). ^^-^^ 
2.2 Approximation errors 

It this section we derive upper bounds for the norm ||P^((5) — Pq'II- These 
bounds produce interpretable sufficient conditions for the proximity of linear 
spaces Uj]-((5) and Ujj". We also present operators that can play a role of the 
main terms of the difference Pj^((5) — Pq- 

2.2.1 Zero perturbation effects 

We start with the necessary and sufficient conditions for the equality Pq{S) = 
Pq. Consider the function B{6) defined in (2.6 1. 

Theorem 2.2. Let (5o > and assume that B{6o) — /iniin/2. Then the following 
assertions are equivalent: 

1. Pif(<5) = P(f forall,5e (-5o,<5o); 
2. 

SoHETpo = SoEETpo = Oll; (2.10) 

3. 

SoB(5)Po = OLi (2.11) 

for all S from a certain neighbourhood of zero; 
4. 



SoHE^Po + PoHE^So 
= SoEE^Po + PoEE^So = Oll; 



(2.12) 



5. 

HE'^Po = Oll and H'^EE^Pq = Olk- (2.13) 
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Remark 2.2. 1. The equality SoEE'^Pq = Oll is equivalent to P^EE'^Pq 
Oll- 



2. Let us discuss the conditions (2.131 



Let U^; stand for the linear space spanned by the columns of the matrix 
E and denote s = dimU^;. Suppose that there exists an orthonormal basis 
Pi, . . . , Ps of the space U^; such that a) each Pi is an eigenvector of the matrix 
EE^, h) Pi,...,Pi e U^, and c) Pi+i, . . . , P,. € Uq for some < I < s. Then 
H^EE'^Po = Olk- (Note that the latter equahty is equivalent to P^EE'^Pq = 
Oll.) 

Indeed, consider Singular Value Decomposition EE"'" — ^21=1 ^i^iPT the 
matrix EE^. Then V^P.^Ql for l<i<l, V^P, = Ol for i > I, and 



P^EETpo^^i.,P^P,(pTp^) 

s 

+ ,.,{P^P,)P^Po = Oll. 



i=l + l 



There are several important particular cases related to the situation under 
discussion. 

If / = s, then Ve C U^, E^Pq = Olx, and both conditions of (|2.13l 



are fulfilled. The example when E is proportional to H (then the equality 
Pf}-((5) = Pq becomes evident) is just a particular case of the inclusion Ve C Uq. 

If / < s, then the natural sufficient condition for the equality HE^'^Pg = Oll 
is HE'^ — Oll- The analogous sufficient condition H"'"E = Okk for the equality 
H"'"EE''"Po — Olk corresponds to the case I — with Ve C Uq. 

Lastly, suppose that s — L and that all singular values of the matrix E 
coincide. If we take I = d and define Pi, . . . , as an orthonormal basis of U[{", 



then we get the second equality of (2.131. This case corresponds to a special 



"noise matrix" E with EE""" proportional to I. 

Consider more explicitly the case where both HE^ and H'^E are zero ma- 



2.2 



trices. Then P;j- [5] = P-^ under the conditions of Theorem 

We call matrices H and E right-orthogonal if HE""" = Oll, or, equivalently, 
EH''' = Oll- If HE^^ = Oll and H'^'E = Ork (the latter equahty means 
that H and E are left- orthogonal), then the matrices are called biorthogonal. 
The biorthogonality condition corresponds to the notion of weak separability in 
Singular Spectrum Analysis (see |13| Sections 1.5 and 6.1]). 

The following statement elucidates the notion of left orthogonality. 

Lemma 2.1. Let H and E be non-zero L x K matrices. 

1. If H and E are left-orthogonal, then G E, PqE = E, and P^E = O^l for 
all non-zero /i S E. 

2. If P^E = Okl for all non-zero /i G E, then H and E are left-orthogonal. 

Remark 2.3. Denote Eh and Ee((5) the sets of positive eigenvalues of the 
operators A = HH"^ and S^E'E'^ = ^^A^^^, respectively. 
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Suppose that H and E are biorthogonal and assume additionally that Eh H 
Ee(<5) — 0. (Note that this corresponds to the strong separability in SSA, see 
|13| §1.5]). Then it is easy to see that SVD of the matrix H + JE is the sum 
of SVDs of the matrices H and (5E. Consequently, both singular values and the 
related singular vectors of the operator H do not change under the perturbation 
H H + (5E. 

Imposing also the condition 5q||EE"^|| = /Ltmin, we obtain that Pj"((5) = P[{" 
for all (5 S ((5o,^o). This condition tells us that each positive eigenvalue of the 
matrix HH""" is greater than all eigenvalues of the matrix (5^ EE""". 

This means that in the case of biorthogonality, we have Pjj- [S) — P^ for all 
5 G (^0, <5o) under the necessary and sufficient condition (5g||EE'^|j — ii^^ rather 
than under the sufficient condition (5q||EE'^|| = /imin/2 of Theorem 



2.2 



If the matrices H and E are right-orthogonal, then B((5) = (^^A^'^'. the 



condition iS^ljEE'^ll < /iinin/2 provides the validity of Theorem 2.1 and, due to 



(2.2 1, the expansion of the perturbed projector takes the form 



_^y^2„ S[,'^U(2)^^ A(2)S[,'''+^^ (2.14) 



p=i ;i+-..+'p+i= 
/,>o 



2.2.2 General upper bounds 



Roughly speaking. Theorem 2.2 shows that under the conditions B{6o) — /imin/2 
and \S\ < So, the equalities SoB((5)Po — and Po"('^) = Pq ^^'^ equivalent. 
Moreover, 

||SoB(.)Po||<M<M<i/2. 

These considerations give rise to the supposition that ||SoB((5)Po|| can serve as 
a natural measure of the proximity Pj]-((5) w 'Pq- 

Theorem 2.3. If > and ||B((5)||//Xmin < 1/4 for all S G (-60,60), then 
where C = ei/V\/^- 



Let us discuss the conditions which provide the right-hand side of (2.151 to 
be small. 

The case of big signal and small noise matrices. 

We start with a condition formulated in terms of eigenvalues of matrices A = 
HH'T and A^^) = EE^. Denote 



0/ ^max ^ ^ Mniax 

/^max /^min 
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where fc'max = IIA*-^-*!!. Note that 0i is a sort of the "noise-to-signal ratio". Since 
02 = ||A|| ||So|| and since Sq is the pseudo-inverse to A, then &2 can be called 
as the "pseudo-condition number" of the matrix A. 



Corollary 2.1. Under the conditions of Theorem |2.3| 



1 



mm 

M,„i„ 1-4||B(5)|1/Mn 



(2.16) 



with 



IB 



Mi, 



Mmin 



(2.17) 



■^0 



Remark 2.4. 1. The inequality (2.16) shows that the difference 'Pq{S) 
is small if the norm of the perturbation operator B((5) is substantially smaller 
than the minimal positive eigenvalue of the matrix HH"'". 

2. Inequalities (2.16), (2.17) jointly yield sufficient conditions for the close 
proximity of the projectors Po{S) and Pq in terms of the eigenvalues of the 
matrices HH""" and EE"'". Roughly speaking, for fixed 6 the difference Pj]"((5) — 
Pq is small if 0i ^ 1 (this means that the signal matrix H is "big" and the 
noise matrix E is "small") and if the positive spectrum of the matrix HH""" is 
not wide-spread in the sense that the quotient Q2 is not very large. 
In particular, if 2\S\ 6162 -I- 9^62 < £ < 1/4, then 



\\P^iS)-P^\\<AC 
The case of the approximate orthogonalities. 



l-4e 



(2.18) 



The upper bound in the inequalities (2.16), (2.17) is rather rough, since it does 
not incorporate the orthogonality properties of matrices H and E. (On the 
other hand, this upper bound shows that P^((5) can be close to Pq without any 
orthogonalities.) Even if these matrices are biorthogonal, still the right-hand 



side of (2.16) remains positive: in this case (2.16) takes a form 



\pus)-pn <^cs^ 



1 



1 - 4(52i/,jj^^//i^ 



To improve this inconvenience we start with the following assertion concern- 
ing minimal principal angles between linear spaces. 

Proposition 2.1. Consider matrices Mi, M2 acting from M^' onto M.^ . Denote 
^min the minimal principal angle between subspaces Ui and U2 , that are spanned 
by the columns of matrices Mi and M2. Lastly, let a^™^'^\ a^^'^^ stand for the 
minimal singular values of Mi, M2, respectively. Then 



(mill) (min) / 

al ' g\ ' cos( 

tT 







< Mi'M2 < Ml M2 



(2.19) 
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The following assertion follows from Theorem 2.3 and Proposition |2.1[ 

Denote 6^ (9i) minimal principal angles between subspaces spanned by rows 
(columns) of matrices H and E. 



Corollary 2.2. Under conditions of Theorem |2.3| 



\P^{5)-V^\\<AC 



|SoB((5)|| 



l-4||B(<5)||//i„ 



and 



|SoB(5)|| < \5\ 6162 (2cos(0,) + \5\ 61 cos(0/)) , 



(2.20) 



(2.21) 



Remark 2.5. 1. Both cos{6r) and cos(6';) serve as proper measures of right 
and left orthogonalities for matrices H and E. 

2. Using inequalities ( 2.20| and (2.211 we obtain sufficient conditions for the 



proximity Pq (i5) ~ Pjj- in the case when the "noise-to-signal ratio" Qi is not 
small. Thus, the "pseudo-condition number" 02 of the matrix HH""" should not 
be very large and both pairs of linear spaces produced by matrices H and E 
should be almost orthogonal. 



2.2.3 Main terms of the approximations 



In this section we discuss special refinements of inequalities (2.151, (2.16), and 



(2.201. More precisely, we present operators that can be considered as the main 
terms of the difference Pj]"((5) — Pj^ in the case when this difference is small 



by norm. The results show that inequalities (2.15), (2.16), and (2.20) produce 



different (though connected with each other) versions of the main terms. 
The case of big signal and small noise matrices. 



Let us start with the inequality (2.16). 
Theorem 2.4. Under conditions of Theorem 12.31 



< 16C 



Wi(,5)| 



ms)\\ 



(2.22) 



l-4||B(J)||//.„ 



where Wi(^) is given by (|2j]) and C = e^l^j^. 

Let us compare (2.22) and (2.16) with the help of the inequality ( 2.17| . If 
B(5)//Xmin is small, then the inequality (2.22) shows that the operator Wi(J) 
can be considered as the main term of the difference P^ ((5) — P J- . In particular, 
||P^(5) - P^ - Wi(<5)|| < II P^ (5) - P^ II if the "noise-to-signal ratio" 61 is 
small and the "pseudo-condition number" 02 is not very large. 

For example, if 2|(5| 9162 + ^"^ Q\<c)2 < e < 1/4, then 



|P^(5)-P^-Wi(J)|| <16C 



1 - 4£ 
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while IIP^(^) - P^ll satisfies ^J^ . 

The case of approximate orthogonalities. 
Let us pass to the upper bound (2.201. 

We start with a simple remark. Denote Aq'^'' — PoA'^^Pq and assume that 
B{So) < /imin- Then it is easy to check that the operator I — S^A^^/fx is 
invertible for any positive /i G S and for any 6 G (— (5o, <^o) since 



,(2) I 



^max ^ B{6q) ^ ^ 



M /^min f-^m'm 

Theorem 2.5. Assume that 5q > 0, B{So) = fi 

PmB(<5)Po 



Li(5) 



E 



/4 and \S\ < Sq. Denote 
1 



and L((5) = Li((5) + Lj{S). Then 

||Po^(<5)-P(^-L(^)||<16cJ 



SpBfflll ||SoB(^)Po| 
l-4||B(<5)||^„i„ 



(2.23) 



(2.24) 



where C = e^l^j^. 

The operator L((5) admits another representation. Denote 



Ki(<5) 



^ p^b(5)a(^) 



M>0 



and put K((5) = Ki((5) + K?'((5). 

Proposition 2.2. Under the conditions and notation of Theorem 2.5 
where Wi((5) is defined in (|2.9|. 



(2.25) 



Let us discuss the result of Theorem 2.5 The upper bound (2.20) is rea- 



sonable in the situation when ||B((5)||//Xniin is not small and ||SoB((5)|| is small 
enough. 

Assume that ||SoB((5)|| w e. Since 

||SoB(,5)Po||<||SoB((5)||, 



then the right-hand side of (2.24) is proportional to while the right-hand 
side of (2.20) is proportional to e. This means that the operator L((5) can be 
considered as the main term of the difference P^ (S) — Pj]- . 

General case. 

Let us study the general upper bound given in Theorem [23] It follows from the 
discussion of Section 2.2.1| that the inequality (2.15) can give weaker conditions 
for the proximity Pq{S) « Pq than inequalities (2.16) and (2.20). 

Let us present a version of the main term of the difference Pq {6) — Pq in 
this general case. 
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Theorem 2.6. Under the conditions of Theorem 12.5 



||p-,.,-P--W,||<«cJ^|||?£-, (2,26, 

where C = e^/V^/^, T((5) = Ti{S) + Tj{S), 

OO 

Ti{S) = J2i-iy J^Li(<5)G„ (2.27) 

i—0 fii,...,fii>0 

J, = Jj(Aii,. . . = riLi (PMfcB((5)/Aife), and 

G, = G,(mi, . . . ,mO = n ~ '^'A^'V^fc 

fc=l 

3 Subspace perturbations for Hankel matrices of 
large size 

Let F — {xq, . . . ,xn^i, . . .) and E = (cq, . . . ,eN-i, . . .). We treat F as a "signal 
series" and E as a "noise series". For fixed N we choose the window length L 
and construct two Hankel ("trajectory") matrices 



H = H 



— Ci-K,L — 



Xo Xi ... XK-2 XK-1 



yXL-1 XL ... XM-2 Xn^i. 



and 

' eo ei ... eK-2 gk-i ' 



E = E 



K,L 



where K — N — L + 1. In terms of Section |2] H serves as a signal matrix and 
E is a noise matrix. 

It is clear that rankH < min(L,X). As it was already mentioned, we are 
interested in the case rank H < min(i, K). To provide this condition, we assume 
that F is governed by a minimal Linear Recurrent Formula of order d. Then 
rankH — d for any L and K such that min(L, K) > d. 

Consider the perturbed series F((5) = F+(5E and construct the corresponding 
Hankel matrix H((5) = H + 5'Ei. Then we can apply all notation and results of 
Section |2] to this particular case of matrices H and E. 

In the present section we derive conditions providing the convergence 
||P^((5) ~ Pffll ^ as ^ cx) and L = L{N). In what follows, we omit 
the dependence of matrices, projectors, etc. on N and L in our notation. 
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3.1 Two general asymptotic results 



We start with two assertions that follow from inequalities (2.161, (2.221 and 



(2.171. Denote 



e = ei 92 = 



Mi, 



Mniax /^min 

Proposition 3.1. If — > as — > oo, then 

'"<8C\S\ 



limsup \\Po{S) 

N 



P-L| 




and 



where C 



limsup Q 

N 



\Pi{S)~Pi-Svi'^\\<C'S' 



(3.1) 



(3.2) 



= 1/6 



and C" = 2(32C + 1). 



Thus we see that 



— for any fixed (5 if 8 tends to zero. This 



means that we can treat O in the same sense as the perturbation parameter S 
for fixed N. Note that 6-^0 only if Qi 0. 

At the same time the condition 9i is not necessary for the convergence 
||P^((5) — Po"|| — > 0. In the next proposition we show that this convergence 
occurs also in the case 8i x 1 if matrices H and E are asymptotically biorthog- 
onal and O2 is bounded from above. (Note that for positive sequences a„ and 
bn we write a„ x 6„ iff ci < an/bn < C2 for some constants ci,C2 > 0.) Denote 
A = l/limsup(eei). 

Proposition 3.2. Suppose that A > 0, 1 1 HE''" 1 1 /^min — > 0, and IISoA^^)! 







as — ^ cx). Then for any 6 with \S\ < Sq 



dcf 



\P^i5)-P^\\ = \5\0{\ 



A/4, 

/Mmin+|<^|||SoA(2)| 



and 



\PoiS) 



L(<S)| 



|HET||//i„,i„ + |5|||SoA(2)| 



(3.3) 



(3.4) 



provided that N > Nq{6). 

Remark 3.1. 1. The operator L((5) is defined in (2.23) 



2. If G01 X 1, then O and Sq < 00. Hence the result of Proposition 3.2 
is valid only if \6\ is bounded from above. These restrictions are absent in the 
case — considered in Proposition |3.1| 

3. Let QQi X 1 and Qi x 1. Consequently, O2 is bounded from above. 
Applying ( |2.21[ ) to ( [33| we obtain that ||P(J-((5) - P^|| ^ if both cos 
cos (6*/) tend to zero. Moreover, (3.3 1 can be rewritten as 



and 



|Pi[(J)-Pi[|| = \S\0 (^cosier) + \S\ COS(0O) 



and the inequality (3.4 1 admits similar reformulation. 
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3.2 Examples. Rough upper bounds 

To illustrate propositions of the previous subsection we consider several types 
of "signal" and "noise" series. For a series F = (/g, ...,/„,.. .) we define the 
associated L x K trajectory matrix in the form 

fa fl ■ ■ ■ /a'-2 /a'-i 



' K,L 



\fL-l /l ■ ■ • fN-2 fN-1 



and denote the maximal and minimal positive eigenvalues of the matrix FF"^ 
as Ainax and Amin, respectively. 

In examples we consider the following four types of series. 

1. A linear combination of increasing exponents. In this case 

/„ = /3iai + (3.5) 



where /3j ^ for 1 < j < p and oi > . . . > Op > 1. The series (3.51 has 
rank p. For short, we name this series the series of exponential type. 

2. A polynomial series. This series is defined by 

fn = Ipn^ + Ip-in^^^ + . . . + 7in + ao, (3.6) 



where 7p 7^ 0. The rank of the series (3.6) is p + 1. 

3. An oscillating series with frequencies wj, positive amplitudes 7; and phases 
(f)i. The series has the form 

p 

/„ = ^7iCOs(27rw;n + (?!)/), (3.7) 
1=1 

where loi G [0, 1/2] and < LUj for I < j. The rank of the series is 2p if 
LJj G (0, 1/2) for all j. If = and ujp < 1/2 (or if wi > and cjp = 1/2), 
then the rank is 2p — 1. If cji = and ujp — 1/2, then the rank is 2p — 2. 

4. A linear stationary random series. By definition, 

00 

/n = ^ Cj (3.8) 



where £„ is the sequence of independent random variables with E£„ = 0, 

De„ — 1, and E|£„|'^ < 00. We assume that S =^ \cj\ < 00 and J2j ^'j — 
1. Then E/„ = and D/„ = 1. Note that the series (/o, . . . , /at, . . .) is 
not a series of finite rank. 

Let us discuss asymptotic properties of eigenvalues Amax and Amin corre- 
sponding to series (3.5)-(3.8) as N 00 and L = L{N). 



16 



Lemma 3.1. 1. For the series of exponential type there exist positive Tm°ax, T'ijn 
such that A„,ax/ar ^ T^L and X^in/al^ xj^l as mm{L,K) ^ c5o. The 
analogous result holds if either L = Lq = const > p or K = Kq — const > p. 

2. For the polynomial series, 

a) if L/N ^ae (0, 1) then Xn.../N^P+^ -> Q^,^ and A„,i„/iV2p+2 ^ e„,i„ 
for some positive Omax, B,„in; 

b) if either L or if is a constant greater than p + 1, then \max/N'^^^^ -> 

©max > and Xniin/N ©min > 0. 

3. For the oscillating series, Xy^ax/LK — > A^ax > and Xmin/ LK — > Amin > 
as min(L, A') — >oo. The analogous result holds if either L = Lq = const > d or 
K = Kq = const > d. 



We omit elementary but laborsome proofs of these assertions. 

Remark 3.2. 1. Positive constants T^^L,T^in^ 9, 
be found in the explicit form and depend both on the parameters of the series 
and on the behavior of L = L{N). 

2. For the oscillating series (3.7 1 we have Amax ^ Amin if L/N a G {0, 1) 



and Amax ^ Amin X if either L or K does not depend on A^. 



3. Parameters aj of the series (3.51 are taken positive only for convenience. All 
results concerning the series of exponential type remain valid under assumption 
|ai| > . . . > |a„|. 



Lemma 3.2. Consider the linear stationary series (3.8 1 



1. If L — >■ oo, then there exists an absolute constant 70 such that almost surely 

2. Assume that L = Lq ~ const and denote • ) the covariance function of 



the series (3.8). Then A^ax/A^ tends almost surely to the maximal eigenvalue 
Umax of the matrix S = {^/(* ~ j)}g<j _i and A„iin/A^ tends almost surely 
to the minimal eigenvalue (Xmin of S. 



Both assertions of Lemma |3.2| follow from results given in Section |7.1| of 
Appendix 2. 



Remark 3.3. 1. The inequality (3.9 1 means that for almost all w and for any 

7' > 7o we get 

AmaxH <7'5iVln7V (3.10) 
provided that A^ > A^o('^, 7')- 

2. li L ~ Lq, then the statement of Lemma [3.2| holds without the condition 
sup„E|£„p < 00. 

3. If L = Lo and /„ — e„ is the "white noise" series, then FF'^/AT — > 1^^, 
Amax/A^ 1, and Amin/A^ 1 a.s. 
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Remark 3.4. Since the series (3.5l-(3.7) are of finite rank, they can serve both 
as "signal series" and as "noise series". Then we use notation Xn or e„ instead 
of /„. In the same manner we use notation /imax or i/max instead of Amax and 
so on. The stationary series can be chosen only as a "noise series". 

3.2.1 Signals of exponential type 



Consider the signal series defined by (3.5 1 and some "noise series" E. The follow- 
ing proposition gives a sufficient condition for the convergence ||Pj- (S) — P(f || — > 
as iV — >■ oo in terms of J^max, oi, and Up. Denote r = ai/a^. 



Proposition 3.3. Let = \l Ti^L / T^^^^. If v^^^t^'^ = o(l) and 5 e M, then 
limsup v^,iL^T-^\\P^{6) - Pq II < 8CSa \5\ and 
limsup ^-.axT-'^||Pc^(<^) - P,^ - SY'iH < C'El6'. 





N 



Remark 3.5. If we take p — 1 in (|3.5[), then /imax = Mmin ^ o^i^ ■ t — a-^^^ < I 



and the condition of Proposition 3.3 reduces to i^max = o(af ) 



Now let us present several examples related to Proposition 3.3 
Example 3.1. 

1. Let e„ = Ym=i^i^? with 7/ 7^ and let b — maxi<;<m Assume that 
6 > 1. Then I'n.i,^ - T^-tLb'^^ ■ If 5t < 1, then for any 6 



limsup (6r)-" ||P^(5) -P^|| < 8CEa\/T^L \S\ and 

N 

limsup (fer)-'^ WmS) -P^- <5VW|| < C'S^T^l d'. 

N 



2. Consider a polynomial "no ise" of order m defined as in ( |3.6| . 

a) In view of Lemma 3.1 t'max ^ ©max^^™^^ in the case L/N — > a £ (0, 1). 
If T < 1 , then for any S 



limsup N-"''\'^ \\P^{S) - P^ll < 8CS,v/©^ l-^l and 
limsup iV-2™-V-^^||P^(<5)-p(-'-5V«||<CS^e_.52_ 

N 

b) If min(L, K) = const, then i^max ^ ©max^^™'*^^- If r < 1, then for any 6 
limsup iV-™-i/2^-w ||p±(^) - P^ll < 8CEay/e^^ \S\ and 

N 

limsup 7V-2™-V-2A'||p^(5)-p(-)-^V«||<CS2e^ax5'. 

N 
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3. Let the oscillating "noise series" be defined as in of (3.7 1. Then z^max ~ 
^maxLK. If T < 1, then for any S 

limsup(iif)-i/V-^||P^(5)-P^|| < 8CEay/A^ \S\ and 

N 

lim sup(Li^)- ||P^ (<5) -P(^ -<5V« || < C'EIA^.^S^ 

N 

4. The case of random stationary "noise series" is studied in Example |3.3| and 
Proposition |3.5[ 



Re mark 3.6. It is easy to see that Proposition |3.3| (and therefore, Proposition 



3.1 1 can give only an upper bound of the true rate of convergence Pf{"((5) — )• Pq. 
For example, for Xn — a" with a > 1 and the constant "noise series" e„ = 1, we 
obtain the equality ||Pj^((5) — Po"|| = 0{Na~^) in the case L ^ K. 



On the other hand. Proposition 4.1 of Section 4.1 affirms that ||Pj-((5) — 

can 



3.1 



'Pq\\ ^ y/Na ^. The reason of this drawback is that Proposition 
ignore the existing asymptotic orthogonalities of signal and noise matrices. 

3.2.2 Oscillating signal series 

Consider the signal of oscillating type with different frequencies ujn G [0, 1/2], 
positive amplitudes ai, and phases (jjn, I = 1, . . . ,p. 

Example 3.2. 

1. Assume that the "noise series" is also oscillating. Namely, let 

m 

e„ = ^ Pj cos(27rWj2n + (f>j2), 

where f3j > and Wji E [0, 1/2]. In addition, let con ^ ojj2 for all l,j. 

If min{L,K) oo, then ^^ax ^ A^^i^LK, ^min ^ a''^.^^LK, i^^ax ~ 
AmaxiiiT, and 69i = ^-max/A^min 1/A =^ Amix/Alii-,,. Consequently, the 



result of Proposition 3.2 is valid for any d such that \d\ < Sq = A/4 provided 
that N is big enough. 

Calculations show that HHE'^ll //!„,;„ = 0(1/ K) and ||SoA(2)|| = 0{1/L). 



Therefore, it follows from (3.3) and (3.4 1 that 



\P^{5)-P^\\ = \5\0{l/inm{L,K)) and 
\P^{S) ~ P^ - L((5)|| = 5^0{l/mmiL\,K^)). 



2. As in Example |3.1| the important case of oscillation signal and random sta- 
tionary "noise series" is studied in Example |3.3| and Proposition |3.5[ 
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3.2.3 Random stationary "noise series" 

Let Sn {n > 0) be a sequence of independent random variables defined on a 
probability space (f2, J^, P). Assume that Ee„ — 0, De„ — 1, and E|e„p < oo. 

In the present section we consider several signal series Xn and the linear 
stationary series e„ as the "noise series". (The series e„ is determined by the 
right-hand side of ( |3.8[ ).) In this case Po"(5) is a random operator dependent on 
w e ri. Moreover, for fixed {6,N,L) the condition (2.11 of Theorem 2.1 holds 
only with a certain probability. 

As in Section |3.2.1| we further present general statements and then consider 
several examples. 

Proposition 3.4. 1. Suppose that L oo. If /imax/Mmin ~ o(l/(iVln A^)), 
then there exist E T with P(rio) = 1 and an absolute constant 7' such that 
for any uj £ Qq and any d 

hmsup , \\F^iS)~P^\\<8C^\S\ and 

N VA^maxA'lnA 

limsup ^Xn Ai W^oi^) '^VW II < SS^ 



where C, C" are defined in Proposition 3.1 and S — J^j l^jl 



2. If L = Lq = const and /imax/Mmin — "^(1/^)1 then there exists a certain 
fl' G jF with P(f7') = 1 such that for any lu E il' and for any 6 

limsup \\P^{S) - P^ll < 8C^^\6\ and 

N V Mmax 

limsup ||P^(,5) - P^ - <5V(i)|| < CVax^', 

where Cmax is the maximal eigenvalue of the matrix S = {i?e(* ~ ■^)}o<i j<l 
and i?e( • ) stands for the covariance function of e„. 

The following examples illustrate Proposition |3.4[ 

Example 3.3. Let N ^ 00. 



1. Consider the exponential signal series (3.5 1 with f3k ^ and decreasing Uk > 1- 
Denote t — ai/Up and suppose that r < 1. 

a) If i — 00. then /imax/Mmin ~ '^'^^ ^.ud almost surely 

limsup (AlnA^)-i/V-^||P^(^) -P^||<8CS,y^|5|, 



N 



lim sup {N In N) - V^^^l | P^ {5) - P^ - <5V|,'^| | < C"~l-ioS5^ 

N 



for any ^. 



b) For the same signal series (3.5 1 and L = Lq = const 

limsup 7V-i/2r-^||P^(5) - P^ll < 8CyW|(5|, 

N 
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limsup N-'t-^^\\fUS) -P^- 5V<,'^|| < C" A„,ax<5^ 

AT 

with probability 1 for any S. 

2. Consider the the polyn omi al signal series ( |3.6[ ) and suppose that L/N 
a G (0,1). Then Lemma 3.1 shows that /imax/Mniin ^ *piV"^^"^ with 
V^Qmax/Qmin- Therefore, almost surely 

limsup (lniV)-i/2^P+°-5||P^((5) -P(j-|| < 8C*p/^l'^l, 



N 



limsup (lniV)-i7V2p+i||p^(5)-P^-5V*,'^|| <C"*^7oS'<52 

AT 

for any S. 

3. Consider the oscillating signal (3.7) and assume that L/N — )■ a € (0, 1). Then 
/^max/^^^ -5> "(I - a)Ainax and finun/N'^ "(1 " a)Amin- Therefore, 

Mmax/Mmin ^ ^^^'^^ ^ o{l/{NlnN)) 

with Tq = (a(l - a)) Vmax/ Amin and 

limsup V^(lnAr)-i||P^(^)-P^|| < 8C T„ Vtc^ l-^l, 



N 



limsup 7V(lniV)-M|P^(5) - P^ - (5V^^ 11 < C'T„*7oS'(52 

N 

with probability 1 for any S. 

The case of oscillating signal series and white-noise "noise series" e„ = £„ is 
of particular importance. 



Proposition 3.5. Consider the oscillating signal series (3.7 1 and i.i.d. "noise 
series" e„ = £„ defined on a probability space (57, J^, P). Assume that Ee„ = 0, 
]D)e„ = 1, and E|£„p < oo. 

If L = Lq = const, Sq < LoAmin/4, and N — > oo, then there exists fl' ^ T 
with P(J7') = 1 such that for any 6 £ {—Sq, Sq) and any uj £ ft' 



limsup A/7V(lnlnAf)-i ||P^((5) - P^ 11 < c' \6\ 

N 



(3.11) 



and 



lim sup A^(ln In A^) - M I P^ ((5) - P^ - T ((5) 1 1 < c"(52 

N 



(3.12) 

where positive constants c' and c" depend on Lq, Sq, and parameters of the 
series (3.7 1. 
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4 Examples. Precise asymptotic results 



The results of Section [3] can provide overestim ated upper bound for the rate of 



pj-i 



convergence 
L((5), and T{S) determined by pJl 



— 0, see Remark 



3.6 



Morover, operators Wi((5), 



(2.231, and (2.271 respectively can serve 



only as candidates for the main term of this convergence. 

In this section we present several examples of "signal" and "noise" series for 
which both true rates of convergence ||Pj]^((5) — Pq\\ — ^ and the related main 
terms are obtained explicitly. 

These examples are rather simple and have minor practical value. In partic- 
ular, we consider signal series of rank 1. Then all relevant operators are derived 
in accessible form. 

In addition, the choice of examples corresponds to different conditions for 
the convergence ||Pj-((5) — P(f || — 0. More precisely, the example of Section 
|4.1| describes the case of "big signal" and "small noise" while the Section [4.2. 1| is 
devoted to asymptotic orthonormalities. The case of white noise "noise series" 
is considered in Section 14.2.21 



4.1 Exponential signal series and constant "noise series" 



WjEj. Then 



Consider the "signal series" Xn — a" with a > 1 and the "noise series" e„ = 1 
Let Wj = (l,a, ...,aJ-i)'^, Ej = e RJ, and A 

\\E,\\ = Vj and IIM^.II = ^{a^^ - l)/ia^ - 1). 

Since matrices H = WlW^ and E = E^E^ have rank 1, then 

Mmin 

Thus, applying Proposition 



El^WEkW = LK and ^i^= n^.r, = WA^WkW' x as iV ^ oo. 



3.1 



with Q 



'L\\ \\yyK\ 

-.N , 



'LK/a' 



(3.2 1, we obtain that for any 5 and N > No{S) 

\\P^{6)-P^\\ = \6\0{VLKa-'') 
and 

||P^(<5) - P^ - ,5VW II = 5^0{LKa-^'' 



and using (3.11 and 
(4.1) 



(4.2) 



where Vq"'^'' is defined in (|2.7l. 



The following proposition presents precise versions of (|4.1|) and (|4.2|). Denote 
H{a,L) 



(a + l) a^^WWLf-Pl 



Proposition 4.1. Let Z^^ stand for EH^So + SoHE^ 

1. liL/N e (0,1), then 



iPoW 



1^1 



a{a + l)^/a^ - 1 



and 



(1)1 



2 (a +1)2 
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2. If K = Kq = const and TV — > cxi, then 



a 



N 



(a+ l)Va2 - 1 



^_„P-(.)-P-HHA_^^-^^ and 

«-||p-(^)-p--^zWH 1^1 

3. If L = Lo = const and iV — > oo, then 

||P^W - Pq II ^ \S\H{a,Lo) while 
||P^(,5)-P^-5VW|| =520(7Va-2A'). 

Remark 4.1. 1. Matrices Vq^-* and Zq^-* have the following explicit form: 
V(^) = Z(^' - Z[,^\ where 

(2) . . W^lW^l 



2. Proposition 4.1 shows that the best rate of convergence of Pi]"(5) to Pq 
is achieved under the choice L = Lq = const. This rate is equal to a^^ and 
the main term of approximation Pq(6) ~ P^ is the linear term ^Vq^''. In this 
case the norms of operators Z^'^ and have the same order of growth and 
therefore we cannot reduce Vq^' to Zq^' as it was done in the case L -> oo. 



4.2 Constant series as a signal 

In this section we consider the constant series = 1 as a signal. Let Wj = 
(1, . . . , 1)"'' e W . Then the L x K trajectory matrix of the series has the form 
H = WlW]^ and ||H|| = V LK. Therefore, d 1 and ^max — Mmin = LK. 



4.2.1 Saw series as a "noise" 

Consider the saw series e„ = ( — 1)" as a "noise". Then E — Ej^Ej^, where 
Ej = (eo, . . . , ej-i)^ . As earlier, let /3j — WjEj. Note that f5j —0 for even j and 
j3j = l for odd j. 



Since t'max — LK, then 0i = 02 = 1 and we cannot apply Proposition 3.1 
vertheless we can use 
the convergence \\Pq[ 
we impose min(L, K) > 1 



Nevertheless we can use Theorem 12. 5 1 and Proposition |3.2| to obtain conditions 
for the convergence ||P|^(5) — P^ll — >■ for small 5 and N ^ oo. Since d = 1 
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Proposition 4.2. Let |^| < 1/2. If L and K are both even, then Pj-((5) = Pq , 
otherwise 



|P(l-W-PcM| = |5| 



K- 



for odd even L 
and iiT — > oo, 
for odd L, even iiT 
and L — >■ cc , 
fori, ii'both odd 
and niin(L, K) — > oo. 

Remark 4.2. Since the series E is periodic with period 2, the different results 
for odd and even L,K are not amazing. In view of |13l ch. 1 §1.6.2], the choice 
of window length as an integer multiple of the period of periodical component 
of the series can essentially improve separability. Here we have got the formal 
affirmation of this principle. 

The case for even L, K exactly corresponds to biorthogonality of matrices H 
and E, see the discussion in Section f2. 2. 1[ Therefore, no additional restrictions 
on L,K are needed. 

Now our goal is to study the main term of the difference PfJ"(<J) 
applying Theorem |2.5| and Proposition |2.2[ 

Consider fixed S such that \S\ < 1/2. Denote 



Pc^by 



El,u, =El-/3l Wl/L, 



(4.3) 



,jWl, and 



We 



LK 

We 



1 - L^U^ 

■ 1 5\ We 



K 



l) ^IJ^ 



Proposition 4.3. The norm of the operator M((5) is 

\5\ 1 



for odd K 
and even L, 
for odd L 
and even 
iorL,K 
both odd. 



(4.4) 



|M((5)| 



= < 



1-6'^K 
1 

1-6^ L 

1-^2 



1 

K 



for odd K and even L, 
for odd L and even K, 
for K and L both odd, 



(4.5) 



while 



|P^(<5)-P^-M(<5)| 



O (L-) 



for odd even L 
and K oo, 
for odd L, even K 
and L — >■ oo, 
fori, Khoth. odd 
and min(L,_R')^oo. 



(4.6) 
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Remark 4.3. 1. Unlike to Proposition 4.1 Proposition 4.3 shows that the 
main term M((5) is nonlinear in 6. 



2. The norm (4.5) indicates the distinctions between positive and negative S in 
the case of odd L, K. For example, if i5 < and L = —6K, then ||P^((5)— Pj^|| x 
whereas this norm has the order for any positive S and L ^ K . 



4.2.2 White noise as the "noise series" 

In this section we study the difference Pj-((5) — Pf{- for the constant signal x„ — 1 
and the white-noise "noise series" e„ = £„ where e„ are i.i.d. random variables 
£„ such that Ee„ = 0, Ee^^ = 1 and Eef^ < oo. 

^The a.s. behavior of ||Pj]-((5) — 'Pq\\ as — )■ oo is already studied in Section 

3.2 Now we derive the main term of the difference Pf{"((S) — Pfi" as iV — )■ oo and 
L — Lq = const in terms of a suitable version of the central limit theorem. 

Denote '^l^ = {ipij}i°jZQ ^ random symmetrical Lq x Lq Toeplitz matrix 
where ipoj {j = 0, . . . , Lq — 1) are independent, i/joo G N(0,Ee'* — 1), and 
ipoj G N(0, 1) for j > 1. (Note that ^ G N(a, cr^) means that £, has a normal 
distribution, E^ = a, and = 0"^.) 

Proposition 4.4. For fixed Lq , S such that (5^ < Lo/4 denote 

rijV = G such that ||B(5)|| < /iinin/2}. 

Then P(f2Ar) ^ 1 as TV oo and 



(4.7) 



where " stands for weak convergence of distributions and C{^) means the 
distribution of the random vector ^. 

Remark 4.4. 1. Roughly speaking. Proposition |4.4| shows that the main term 
of the difference Pfj"((5) — Pq has the asymptotic form 

(Pf^M/^^Po + PoVl/^^P^). 



LoVN 

2. Since Lq is fixed and the norm || • || is the continuous functional on the space 
of Lq X Lq matrices, then 



c(VN\\p^iS)-p^\\\n 



N 



^( £||Pc^*Lo Po + Po*Lo Pol 



under conditions of Proposition 4.4 
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5 On the way to applications 



In the present section we briefly describe the appHcation of the previous results 



to several methods of Signal Subspace Analysis. Section 5.1 is devoted to the 
approximation of linear recurrent formulas governing the signal and in Section 
5.2|we study the real- valued variant of Least-Square ESPRIT. We demonstrate 



that both methods asymptotically converge under assumption that ||Pq ((5) 



In Section [5T2| we discuss the reconstruction stage of Singular Spectrum Anal- 
ysis (briefly, SSA). Though the precision of SSA can not be described only in 
terms of proximity of the perturbed and unperturbed projectors, the obtained 
results help to reformulate the problem in a more transparent form. 



5.1 Approximations of linear recurrent formulas 

Let the signal series F = (xq, . . . , . . .) be governed by a linear recurrent 
formula (LRF) 



d 



c„ = ^5fcx„_fe, n>d 



(5.1) 



fc=i 



and suppose that (5.11 is the minimal LRF for the series F. In particular, this 
means that bd ^ 0. 

For L, if > d let H stand for the trajectory L y. K matrix of the series 
F. Then rankH = d. Consider the signal subspace UfJ- and the corresponding 
projector P^j-. Denote cl = (0, 0, . . . , 0, 1)""" S M^. As it is proved in [TS] sect. 
5.2], PoCl 7^ 0. (Note that ||PoCl|| is a cosine between vector and linear 
space Uq.) 

Let Gl stand for the (L — 1) x L matrix 



Gl = 

and denote 

R = (aL_i, . . 
Due to [H th. 5.2], 



/I 


. 


. 





0\ 





1 . 


. 











. 


. 1 










. 


. 


1 


0/ 



(5.2) 



L-l 



IPoClI 



GiPoCi. 



^afca;„_fc, n>L. 



(5.3) 



(5.4) 



k=l 
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Note that analogous expressions are known from early 80s, see for example, 



It is clear that (5.4 1 coincides with (5.1 1 for L — d + 1 



The formula (5.3) can be used to derive the approximation of R in the case 
when the signal series F is corrupted by an additive noise series E. In other 
words, if F((5) = F + 5E, then the natural form of this approximation has the 
form 



m = - 



1 



GlPo{S)cl- 



If Po(i5) is close to Po, then R{S) must be close to R. 

Proposition 5.1. Let AP{6) stand fo r \\P^{ S) - P^||. Denote ■& = 
and suppose that AP((5) < ||PoeL|| = Vl - Then 



fill 



(5.5) 



Proposition |5 . 1 [ gives sufficient conditions for the convergence ||i?((5)— 
in terms of ||P^) - P^|| and ||PoeL||. Namely, if \\P^i5) - P^|| ^ and if 
llPoeill is separated from zero, then ||i?((5) — R\\ 0. 

The second condition automatically holds in the case L = const > d (then 
both and Pq do not depend on N). It is not difficult to show (we omit 
proofs for brevity) that ||PoeL|j is separated from zero as L — ^ oo for signals of 
exponential type, polynomial signals and oscillating signals. Therefore, in these 
cases takes the form \\R{5) - =0(||P^((5)-P^||). Examples of Section 
|3.2| refine this general assertion for a number of "signal" and "noise" series. 



5.2 LS-ESPRIT for real-valued signals 



As in Section 5.1 we suppose that the signal series F = {xq, . . . , Xn, ■ ■ ■) is 
governed by the linear recurrent formula (5.1 1 with bd ^ 0- It is well-known that 



the general solution of (5.1 ) is expressed through the roots of the characteristic 

polynomial Pd{\) = Xd - ELi bkX'^~^ . 

Assume that L,K > d and consider the L x K trajectory matrix H of the 
series F. As in previous sections, let Vq stand for the linear space spanned 
by columns of the matrix H. For a certain basis Ui,...,Ud of Vq denote 
U = [Ui : . . . : Ud]- Lastly, let the matrix Gl be defcred by (5.2 1 and denote 
the (L-l) X L matrix G^^^ by 



G(^) 



/o 


1 


. 


. 


^\ 








1 . 


. 











. 


. 1 










. 


. 


1/ 
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Note that 



/I 



0\ 




1 



Fi 



del 




and 



... 
\0 . . . 

/O 1 
1 



1 
0/ 



0\ 




F2 



del 






yo 



1 
0/ 



(Both matrices are of the size L x L.) It is easy to see that ||Fi|| ~ IIF2II ~ 1. 

The method called Least Square (briefly, LS) ESPRIT for the analysis of 
the series F is based on the following facts (see discussion in [23] and references 
within) : for any basis Ui, . . . ,Ud of the linear space Vq 

1. the matrix U'^FiU is invertible; 

2. the set of eigenvalues of the matrix 



coincides with the set of roots of the polynomial fd(A) subject to multi- 
plicities of roots and eigenvalues. 

In practice, leading left singular vectors of the matrix H usually stand for the 
basis Ui, . . . ,Ud of linear space Ufj". The similar choice gives rise to approxima- 
tion of the matrix D in the case when the series F is corrupted by an additive 
"noise series" E multiplied by a formal perturbation parameter 6. 

Denote E the Lx K trajectory matrix of the series E. If Uj (5) {j = 1, . . . ,d) 
are leading left singular vectors of the matrix H((5) = H -I- (5E and U((5) = 
[Ui{6) : . . . : Ud{S)], then we can use the matrix 



to approximate D. 

The following assertion helps to express D((5) through the perturbed projec- 



Lemma 5.1. Let Ui, . . . ,Ud be a basis of a linear space U C M^. Denote P 
the orthogonal projector on U. 

1. If Q : i-^ and ||Q - Pj] < 1, then vectors QC/i, . . . , QUd are linearly 
independent. 

2. Consider a linear space V C of dimension d and denote Q the orthogonal 
projector on V. If ||Q — P|| < 1, then there exist linearly independent vectors 
Vi, . . . , Vd e ¥ such that Uj = PVj. 




(5.6) 




(5.7) 



tor 



(S). 
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Lemma 5.1 shows that if |1P^((5) — Pj-|| < 1, then the matrix 



(5.8) 



where U = [Ui , . . . , Ud] , has the same eigenvalues as the matrix (|5.7| for any 
choice of the basis Ui, . . . ,Ud of the Hnear space Vq. Moreover D((5) = 'D{S) 
under a certain choice of Ui, . . . ,Ud- 

Suppose now that \\P^{S)-P^\\->-0 as N = K+L-l->-oo. Then it is natural 
to suppose that ||D((5) — D|| — > and therefore asymptotically as — cx) we 
get all roots of the polynomial i-d(A) at least in the case L — const. 

Let us formalize these considerations. As earlier, consider a certain basis 
Ui, . . . ,Ud of the linear space and set U = [Ui, . . . , Ud\- 

Proposition 5.2. If AP{S) =^ \\P^{S) - P^\\ < v/2 with v standing for 
||UTFiU||/||Uf , then 



|D((5)-D|| < 



2AP(<5) 



1 



1 - 2AP((5)/w 



(5.9) 



Remark 5.1. Denote = (0, 0, . . . , 0, 1)'^ e and = ||P^eL||. Then 

Fi = I — tL^^L 

U^FiU = U'^U - U^CicJU 
-UTU-UT(p^ei)(elPf^)U. 

Therefore, ||UTFiU|| > ||U||2 (l -i?2^ and under restriction AP{6) < {l-d'^)/2 
the inequality (5.9) can be transformed to 



|D(,5)-D||< 



2AP(,5) 

1-^92 



1 



1-2AP((5)/(1-?$|2) 



(5.10) 



Note that the upper bound (5.10 1 does not depend on a basis of the linear space 



The inequality (5.101 shows that ||D((5) — D | — > under the same conditions 

More precisely, \\P^{S) - P^\\ 



5.1 



as for linear recurrent formulas of Section 

must tend to zero and IIPoClIP = 1 — niust be separated from zero. 

Moreover, in this case ||D(<5)-D|| = 0(||P^((5)-P^||). Therefore, examples 



of Section 



3.2 



provide the corresponding upper bounds for ||D((5) — D| 



5.3 On the reconstruction stage of Singular Spectrum 
Analysis 

If the aim of Singular Spectrum Analysis is treated as an interpretable decompo- 
sition of time series Gn — (go, ■ ■ ■ tGn-i) onto 2 or more additive components, 
then the whole SSA procedure can be expressed as follows. (See |13l ch. 1] for 
details; for our goals it is sufficient to decompose Gat onto 2 components.) 
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The decomposition stage consists of the choice of the 'Svindow length" L, 
construction of L x K "trajectory matrix" Gn of the series Gn and Singular 
Value Decomposition of G onto "elementary" rank-one matrices G^^\ 

The reconstruction stage consists of the summation of a certain number of 
G^'' (then we get the "reconstructed" matrix G'^) and "hankelization" of G^. 
Formally, the result SG'j^ of hankelization of the matrix G'^ = {ffijl^g^ j=o^^ 
is the L X K Hankel matrix with elements g'^j equal to the average of such 
that k + I = i + j. Since each L x K Hankel matrix is in natural one-to-one 
correspondence with a series of length N = L + K ~ 1, then we obtain the 
decomposition Gn — G'^ -I- {Gn — G'^) of the initial series Gjv- 

Suppose now that the series Gat is the sum of the "signal" Fn governed by 



LRF (5.1) with bd ^ and the series 5F,n, where Eat is a "noise series" and S is 
a formal perturbation parameter. Then it is natural to state the problem of an 
(approximate) extraction of the signal Fjy from the sum G^r = F^v + SEn- 

For small S the standard approach to this problem is expressed in terms 
discussed in previous sections. Namely, under the choice oid<L<N — d~l 
the trajectory matrix Gn = H(i5) = H-f i5E is processed by SVD and d leading 
elementary matrices are summed to get the approximation H — G^ of H. Then 
the hankelization procedure yields the approximation Fat ((5) of the series Fn- 

Note that H = P^H. Then H = P(J-((5)(H + JE) and therefore the approx- 
imation error A5(H) = H — H of the reconstructed trajectory matrix has the 
form 

A,(H)-Pt[(^)H(5)-P^H 
= (Pf^(<5)-P^)H(5) + JP^E. 



Remark 5.2. It follows from results of Section 
H and E are biorthogonal. 



2.2.1 



that H = H provided that 



To measure the difference between F n (<5) and F^r we must introduce a con- 
venient metric. In many practical cases the proper choice is 

Assume now that for any N both F^v and Ejv are segments of infinite series 
F and E. If ||FAr(5) -F7v||max tends to zero as L ^ L{N) and iV 00, then SSA 
asymptotically reconstructs the (infinite) series F from the (infinite) perturbed 
series F -|- (5E. It is easy to see that 

\\Fn{S) - F^||_ = ||5(P^(<5)H(<5)) - H||_ 
= II'5A.(H)||_, 

where ||A||niax = max^j \aij\ for the matrix A with entries aij and S stands for 
the hankelization operator. It is clear that ||5A||inax < ||A||max- 

The spectral norm ||A|| and uniform norm ||A||niax are equivalent, but this 
equivalence is lost as the size of a matrix tends to infinity. 
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It is well-known that ||A|l„iax < |1A|1. This means that if the spectral norm 
of the matrix A is small, then all entries of the Hankel matrix SA are small too. 
The opposite inequality, which is also well-known (see |25] for both inequalities) 
has the form ||A|| < V Li^|| A||niax for an i x if matrix A. This inequality 
gives a hint that a large-size matrix with small entries can have a large spectral 
norm. Indeed, the nxn Hankel matrix G„ with equal entries g^J-* =n~^/^ has 

the spectral norm ||G„|| = \fn — >oo while g^^"''— >0 as n— >oo. 

In general, this means that even if the spectral norm || (H) 
to infinity ?& N ^ oo, still the convergence ||Fjv((5) — F^vUmax 
Therefore, talcing into account the last term of the right-hand side of (5.11 1, we 



does not tend 
->■ can occur. 



see that general upper bounds of the kind (5.5) or (5.9) can hardy be valid for 
l|F^('5) 



However, equalities (5.11) and (5.12) help to simplify the problem. Suppose 



that 



||(P^(5)-P^-N(5)) H(J)||^0 
as — ^ cx) for certain operators N((5) = Nat (5). Then 

5A,(H) = S{{V^{b) -Vi- N(<5))H(<5)) 



(5.13) 



-5(N((5)H(<5) -H(5P^E) 



and 



|FAr(<5) 



|5(N(5)H((5)+(5P([E)| 



o(l) 



as iV — >■ oo. Thus, the problem reduces to the investigation of the asymptotic 
behavior of entries of Hankel matrices 5(N((5)H((5) + (5Pj]^E). 

Let us consider several examples of signal and noise series when the con- 



vergence (5.13) occurs. In all examples we straightforwardly apply inequali- 



ties of Section |3.2[ therefore results are given without comments. The norm 
||(Pj-(^) - P^ - N((5)) H((5)|| is estimated as 

||(Pf^(<5)-Po^-N(5))H(<5)|| 

< ||Pf^(<5)-Po^-N(<5)|| (||H|| + |<5|||E||). 

In the most of examples N((5) = (^Vq^-*. Due to (2.7), we have 

V*,^^H(,5) = PqEH'^SoH + (5(PoEH'^SoE -f SoHE^PqE). 

For short, denote 

A(N) = N(<5)H(J) + ^Pt[E and A(N) = ||5A(N)||_ . 



Example 5.1. Signals of exponential type. (See Example |3.l| of Section 3.2.1 ) 
Consider the signal series (3.5) with (3k^0 and decreasing at >1. Then ||H|| x 

3/2 



,7V 



Denote 9 = 



la 
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1. Let the "noise series" be defined by e„ — X^/Hi ^i^? with 7/^0 and let 
16,1 > 1. Thus IIEII X 6^. If b0 < 1, then llEll = o(||H||), 



, def 

b = maxi</<j 



|P^(<S)-P^-5VW||||H(<5)|| =0(6^?) 



\2N 



for large N, and 



|Fjv(<5) 



'N 



A(JVW) + O ((60)2^) 



2. Consider a polynomial "noise" of order m defined in the manner of (3.6 1. 



a) If L/N ^ a e (0, 1), then ||E|| 



= o(||H||). If < 1, then 



b) If TV 00 and either L or K is a constant, then ||E|| x 7V™+i/2 = o(||H||). 
If 6* < 1, then 



\FNiS) 



'N 



0{N 



2m+ln2N\ 



3. Let the oscillating "noise series" be defined in the manner of (3.7 1. Then 
||E|| X VLK = o(||H||). If 6* < 1, then 



I Fat (,5) 



' N 



2N\ 



(5.14) 



Example 5.2. Oscillating signals. (See Example 3.2 of Section 3.2.2[ ) 
Let the oscillating signal be defined by (3.7). Then ||H|| x \JLK. If the "noise 
series" is also oscillating but with frequencies different from those of a signal, 
then IIEII x VZif x ||H|| and 



|Fjv((5) 



A(L((5)) +0(V£if/min(L2,X2)) 



under assumption that \b\ < Sq for some Sq > 0. Note that the operator L((5) is 
defined by (lOsl. 



Example 5.3. Random stationary "noise" series. (See Example 3.3 and propo- 
sition 



3.4 3.5 of Section 3.2.2 ) 



1. For a signal series of exponential type and under related conditions the fol- 
lowing results hold. 

a) If TV — )• 00 and L — > 00, then almost surely ||E||/\/7V In TV < c for > 
Nq{uj) and some constant c > 0. Therefore, ||E|| = o(||H||) with probability 1. 
If 6* < 1, then 



|Fw(<5) 



\{6Y^o^) +0{N\nN9 



2N\ 



almost surely for > Nq{uj,S). 

h) li L ~ Lq — const, then ||E||/\/]V tends a.s. to the positive constant cq 
and 



with probability 1 for N > No{uj, S). 
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2. Let a polynomial signal of order p be defined by (3.6 1. If L/N — >■ a £ (0, 1), 
then ||H|1 x Np+'^ while ||E|| has a.s. the order of growth VNlnN. Therefore, 

\\Fn{6) - F^||^_ = A(<5V«) + O {InN/Nn 

almost surely for > No{uj, S). 

3. Consider the oscillating sig nal defined by (3.7 1 and the white-noise "noise 
series" as in Proposition 3.5 li L = Lq = const, then ||H|| x ^/N and 
||E|j/V]V 1 a.s. Thus ||H((5)|j = 0{Vn) with probability 1. Since almost 
surely 

limsup N{\nlnN)-^ \\Poi^) -Pq- T(<5)|| < c"S^ 

N>Noiui) 

with a positive constant c". then 

||Fjv(<5) - Fn\\^^^ = A(T(<5)) + 0{hilnN/VN) 

almost surely for > Nq{uj,S) and \S\ < Sq- 

Let us present two simple examples where these considerations are followed 
to the end. 



5.3.1 Reconstruction: constant signal and saw "noise series" 

Consider the signal a;„ = 1 and the "noise" e„ = (—1)". This example was 
studied in detail in Section 4.2.1 where the main term ]V[((5) of the difference 
Pj^((5) — Pq was derived in explicit form, see (4.4|, under condition \S\ < 1/2. 
Here we use results and notation of this section. 

Proposition 5.3. 1. If both L and K are even, then Fjv((5) = F^- 
2. If L is even, K is odd, K oo and L = o{K^), then 



l|F^(^)-F^Lax = 0(max(i^-i 

3. If K is even, L is odd, L ^ oo and K = o{L^). then 

\\Pn{5) - FjvL,,, - O (max {L-\ ^kJI 

4. If both L and K are odd, L = o{K'^) and K = o(L^), then 

\\^n{6) - FjvL^^^ = O (min {^/lJk^, ^kJiJ) 



(5.15) 



(5.16) 



(5.17) 



Remark 5.3. The best order of convergence to zero of the right-hand side of 
(5.151 is N^^. It is achieved under the choice L = Lo = const. The same result 
for (5.161 is attained by the choice K = Kq and for (5.171 by the choice L^N/2. 
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5.3.2 Reconstruction: exponential signal series and constant "noise 
series" 



Consider the signal series a;„ — dP- with a > 1 and the "noise series" e„ = 1. 
Then B = ^J~a and applying (5.141 we see that 



in the case LjN — > a G (0, 1) and that 



in the case when — > oo and either L or X is a constant. 

Proposition 5.4. If L/A^ a e (0, 1) or if cx) and either L or is a 

constant, then 

limsup||FAr((S)-Fjv||^^^^ >0. 

AT 

Remark 5.4. The accurate analysis of the matrix 5Ai((5Vq^-') (we omit cal- 
culations due to their technical character) gives much more information on the 
behavior of the series Fjv(i5) as A^ oo. Let us consider the simplest case of 
odd A^ with L = K. Denote 6 = (a + l)/(a - 1) and A/„(,5) = /„((5) - ji(S). 
Then the following asymptotic results hold. 

1. If / < L, then 



max 

o<;<L 



A/, (5) -2,56 ■ 



= |^|0(a-2^) (5.18) 



as L ^ oo. In particular, A/o(5) = 25(a + l)a"^ + 0(a"^^), A/l_i((5) 
25 6L-\ and 

A/,(<5)~256-^a-(i-^o)^ 

in the case l/L ^ (0, 1). 

2. Ifl = 2L-k> L, then 



max 

l<k<L 



Af2L-kiS)-2db 
\S\0{La-^) 



a(a'^-i-l)-(a2-l)(fc-l) 



(fc - l)a'= 



as L — > oo. This means that for any small e > 

1 



max 

le[L, (2~e)L] 



Afii5)~2Sb 



\5\0{L-'). 



In particular, if l/L ^ Aq G [1, 2), then iA//^^ ^ 2(56(2 - Aq). 



(5.19) 
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These results improve the accuracy of Proposition 5.4 in the case L — K. 
Namely, though the sequence maxo<;<Ar \fi{5) — fi\ does not tend to zero, still 
maxo<«(i-e)jv \ fi{S) - /;| as -> oo for any small e. 

Computer experiments confirm these considerations. For example, two lines 
of Fig. [l] present SSA-computed reconstruction errors A/„ {6) for the signal 
fn = (1.01)" with the noise series Cn = 1 and 6 = 1. Red line corresponds 
to TV = 999 and L = K = 500, blue line describes the case N = 1999 and 
L = K = 1000. 
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Figure 1: Errors of reconstruction. 



Both curves illustrate results of Remark |5.4[ In particular, analytical main 



terms of the differences A/„(i5) presented in (5.181 and (5.191 perfectly corre- 
spond to the computed data of Fig. [l] 



6 Appendix 1: Proofs 



Proof of Theorem 



1. The equivalence between (2.101 and (2.111. Since 



SoB(5)Po = 5 SoHE^Fo + 5^ SqEETPq, 



then (2.11 1 automatically follows from (2.10 1. On the other hand, if SoHE'^'Po- 
6 SqEE^ Pq = for all 5 € (— then we obtain the equalities (2.101. 



2. The equivalence between (2.101 and (2.121. It is easy to see that (2.121 



follows from (2.101. Let us demonstrate that the equality SoHE'^Pq = ^ll 



follows from the equality 



SoHETpo + PoHE^So-Oii 



(6.1) 
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and that the equality SqEE^'^Po — Oll is the consequence of SqEE^'^Po + 
PqEE^'^So — Oll- Both assertions have similar proofs. Suppose that (6.11 
holds and denote X = HE^PoZ and Y = HE^So^ for some Z G 

equality SqX + PqY 

the latter vectors are orthogonal. Thus SpHE'^'PoZ 
Z G M^. 
3 



Then the 

Oi, follows from ([el). Since SqX G and Po^ G Uq, 

PoHETSqZ = Ol for all 



The equivalence between (2.101 and the equality Pq(S) 



Pq . Assume 

that (|2.10[) holds. Then SoB((5)Po = PoB((5)So = Oll- Consider the operator 



Sr'B(5)S<, 



iiip) 



B(<5)S^' 



(/p4 



(6.2) 



see (2.3). Since li + . . . + /p+i = p, then there exists j such that Ij > and 
either Ij-i— or Ij+i—O. Suppose that Zj+i=0. (The case lj-i=0 is studied 



analogously.) Since Sq 



(0) 



^0 ■ 



-Po and S[,'^^ 



then s|,'^^B(5)So 



and 



the operator ( |6.2| is the zero operator. Hence Wp((5) = Oll for all p. and 



In order to prove the converse assertion we consider the expansion (2.4 1. 
Since Pij"(<5) — Pq for all S G {—Sq, Sq), then the coefficients Vq"-* of the power 



series ( |2.4[ ) are zero operators. In particular. Vq^-* = Oll and (6.1 1 holds due to 
(2.7 1. Let us consider the coefficient Vq^'. As it is already proved, SqA'^^-'Po = 
PoA(i)So = Oll- Thus ^ shows that Oll = V^^^ = PoA(2)So + SoA(2)Po, 



and we get (2.12). The latter equality is equivalent to (2.10). 

4. The equivalence between (2.10) and (2.13). Let us prove that the equality 



HE^Po = Oll follows from SoHE^Po = Oll- (The converse statement holds 
automatically.) Suppose that SqHE^^Po = Oll and let Z e M.^. Note that 
vectors P^HE'^Pq.^ are pairwise orthogonal for different /i G S. Therefore, 



SoHE^PoZ = /^"' P^HE^Po^- = Ol, 

l-i>0 

then P^HE"^Po^ ~ Ol for all non-zero ^ G S. In view of the equality H = 
Y.^>o we see that HE'^Pq = 0. 

Let us prove that the equahty H'^EE'^Po = Olk holds iff SqEE'^Po = Oll- 
If H'^EE^Po = Olk, then 

Oll = HH^EE^Po = ^ a^P^EE^Po, 

M>0 



P^EE'^Po = Oll for all non-zero fj, G E, and SoEE'^Pq = Oll- The converse 
statement is proved in the same manner due to the equality H'^^EE^'^Pq = 
^^j.g H'^P^EE'^Pq. This completes the entire proof. 

Proof of Lemma 12.11 

1. Let H^E = Okk- Then H^HEX = Ol for any X G M^'. If ^ S, then 
EX = Ol and E = Okl- Therefore, G E and EAT G Uo. Thus PqEA = EA 
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and PoE = E. Analogously, P^E = P^PoE = Okl for all non-zero /i e S. 



2. Since H = E^>oPmH, then H^E = (^Ep>o P^Ej , and the second 
assertion is proved as well. 

Proof of Theorem [23] 

Let us start with the following lemma. 

Lemma 6.1. If < ^ < 1/4 and k>0, then 



E 

p—k 



<c 



1-4:13 



•3) 



where C = e^l^^j^ « 0.667. 

Prool Since n\ = V2^?i"+^/^e-" e<^r.l\in ^-^.^ g < 6I„ < 1, then 



and the result becomes clear. 



□ 



Now let us demonstrate Theorem 2^ Due to Theorem |2.1| we can use the 
expansion (2.2 1. 

In the same way as in the demonstration of Theorem |2.2| consider the term 



(6.2 1 and take j such that Ij > and = (the case Ij^i = is quite 

analogous). Since li + . . . + Ip+i = p and ||Sq'^''|| = l/^J-aiin f*^^' any fc > 0, then 
the norm of the operator (16. 21) can be estimated from above: 



S('^^B(,5)... S^'^-^^SoB(<5)Po...B(5)Sj,' 



(6.4) 



< ||SoB(5)Po|| {\\B{S)\\/fi„ 



Denote 3p the number of nonnegative integer vectors (/i, . . . , ^p+i) such that 



li + ... + Ip+i = p. Then % = . In view of (|2^ and ( |6^ 
\\Po{S)-P^ 



< ||soB(5)Po|| J2 E (iib(^)ii/m. 

p=l ii + ...+/p+i=p, tj>0 

= ||SoB(5)Po|| E 



(I|B(<5)||/a.„ 



.p-i 



Now we get the result with the help of (6.3 1. 
Proof of Corollary 
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Since |lSoB((5)Po|l < ||B((5)||//iijiin, then (2.161 automatically follows from 
(2.151. The inequality ||B((5)||//i„iin < B{5)/ii^in was already established. The 
chain of inequalities 

m5)\\ < < 2|^| I|HET|| 



Mmiii Mmin Mmin Mmin 

< 2\5\ HM + 52 9202 ^ 2|5| 6162 + <5^e?e2 

provides the result. 

Proof of Proposition |2.1| 

We begin with the following lemma. 

Lemma 6.2. Consider the matrix M : M^' i-> and let Ua/ stand for the 
linear space spanned by columns of the matrix M. Denote dmin and dmax the 
minimal and maximal singular values of M. Then 

{MX, ||x|| < i/<7,nax} c {r e Um, \\y\\ < 1} (6.5) 

and 

{MX, ||X|| < l/a,ni„} d{Ye Um, \\Y\\ < l}. (6.6) 



Proof. The inclusion (6.5 1 straightforwardly follows from the inequality 
||MX|| < ||M|| ||X|| < 1. Let us prove the inclusion 

Consider the SVD of the matrix M: M = J2j (^jUj . By definition, left 
singular vectors Uj form the orthonormal basis of Um- (Analogously, right 
singular vectors Vj form the orthonormal basis of the linear space spanned by 
rows of the same matrix.) 

Let us prove that for any F G Ua/ with ||y|| < 1 there exists X e M?'" such 
that ||X|| < l/(Tmin and MX = Y. Then will be proved as well. 

Note that Y = "Y^- CjUj with J^j c| < 1- In the same manner, each X G M.^ 
can be expressed as X = oij^j + PqWq, where (V,, Wo)^ = and 
stands for the inner product in . Therefore, MX = J2j '^j'^j^j- 1^ '^^ P^^ 



a 



Cj/aj, then J2j < J2j ^j/'^min < 1/^min *ind the proof is complete. □ 
Now we turn to the demonstration of Proposition |2.1[ By definition, 
cos(6'min)= max max (1^1,12),- 

neOi,||Yi||<l Y2G02,||Y2|1<1 ^ 

In view of Lemma |6.2| 

IIM1IIIIM2II COs(0,ni„) 

< max max (MiXi,M2X2)r 

XiGR^,||Xi||<i Jf2eR-^,il-Y2||<i 

= max max (M^M2Xi,X2)„ 

XieK^,||Xi||<i X2eR-f^,||X2||<i " 

= ||mJm2||. 



38 



If we use (6.61 instead of (6.5 1, then we get 



(min) (min) /p. \ 

a\ ' a\ cos(6l„ii„) 

> max max (Mi^i, M2X2) , 

XieR«,||Xil|<l X2GR-^,||X2i|<l 



= llM^'Mall. 
This completes the proof. 
Proof of Corollary 



It is easy to see that (2.20) follows from (2.151. Besides 



||SoB(5)|| <2|(5|||HET||/^,„i„ + <52||SoA(2)||. 
Proposition |2.1| shows that 



(6.7) 



lHET||/^,„i„ < cos(0,)Va^ 



max'^max/ pmin 



6182 cos(0J 



As for the term ||SoA'^^)||, it is worth to mention that the columns of the 
matrix So = Sj span the same linear space \]q as the columns of the matrix H. 
Analogously, the columns of A*^^^ span the same linear space as the columns of E. 
Therefore, it follows from Proposition 2.1 that ||SoA'^^)|| < cos(6';)i'niax/Aimin = 
0^82 cos(6';), and the proof is complete. 



Proof of Theorem 12.41 

Acting in the same manner as in Theorem |2.3| and taking into consideration 
that ||SoB((5)|| < ||B((5)||//j,iuin we get the inequality 



p=2 



Applying Lemma 6.1 and the condition ||B((5)|| < /imin/4 we obtain (2.22) 
Proof of Theorem 



First of all, we can use the decomposition (2.2). Let us extract the operator 



L((5) from the right-hand side of (2.2) 



For fixed p > 1 we take the vector (Zi, . . . , Zp+i) with Iq ~ p and Ij — for 
j > 1 and consider the related operator 

Xi(p) = (-1)PS^'^^B(^)S^'^'... B((5)S^'^+^^ 

It is clear that 

= % B((5)Po . . . B(5)Po = B(<5)Po (PoB(5)Po 
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Since PoB(5)Po = ^^A^,'^ and = Em>oPp/a*^ then 

Xi(p) = ^^-?'p^B(5)Po [s^aI; 

fj.>0 



p-i 



M>0 



Thus we obtain that 



p=i 



ti>0 



M>0 

If we consider X2(p) = (-1)^ s[,°^B(,5)s[,°^ . . B((5)s[|''^ and act in the same 
manner as for Xi(p), we get that 

ThusL(p)=Ep>i(Xi(p)+X2b)). 

Now consider the vector [li, . . . , Ip+i) such that = -P neither 

?! nor /p+i equals p. (It follows from this condition that p > 1.) In addition, 
consider 1 < j < p + 1 such that Ij > and either Ij^i or Ij^i equals zero. If 
Ij = p and 1 < j < p + 1, then 

||S^°^B((5) . . . B(5)S^^)b(5) . . . B((5)S[,"^ II 
= ||PoB(5) . . . PoB((5)SoSr'SoB(,5)Po . . . B(<5)Por 



<||PoB((5)So|| ||SoB(<5)Po|| ||Sg-^|| 
<||SoB(<5)Po|| ||SoB(5)|| (|1B(<5)||^„ 



(6.8) 



\P-2 



If Ij 7^ p, then there exists k ^ j such that Ik and we get the same upper 
bound (6.8 1. For example, let 1 < < < p + 1 and /j+i = 0. Then 



S^'^^B((5)...B((5)S^'^^.. 



s[,'^'b((5)...B(5)s[,'''+^^ 



*0 



SoB(^)Po . . . S^* 



< ||SoB(,5)|| ||SoB(5)Po|| ||Bf"2 ||Sp||'i+-+;p+i-2 
= ||SoB(<5)|| ||SoB(,5)Po|| {\\B{6)\\/fi^inY'\ 
Other variants concerning fc,j provide the same result. Therefore, applying 
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(2.2 1 and Lemma 6.1 we obtain that 



|Pi[(<5)-P^-L(5)| 



p=2 ;i+...+ip+i=p 

lj>(), h^p, Ip+i^p 



<^ ^ ||SoB(<5)|l|lSoB(5)Pc 

p=2 /i + ...+ip+i=p 
ij>0, ii^'tp, Ip+i^p 

oo 

= |iSoB(5)|i||SoB(<5)Po||E 

p=2 



< ||SoB(<5)||||SoB(5)Po||E 
|SoB(5)||||SoB(<5)Po| 



2p\ /||B(5)| 



|B(^)|| 



1B(^)||\^ 



p-2 



p-2 



< 16C 



1-4|1B(<5)||/Mn 

The proof is complete. 



Proof of Proposition 2.2 

First of all we have 



Therefore, we get that 

L(5) = ^ (PpB(^)Po + PoB(,5)P^)/a* 

M>0 



^2 A (2) , 



M>0 ^ \ / 



5: (i-^^a^^Va^)" -I 

At>0 ^ 



= (5 So (HE'^ + EH^) Po + <5 Po (HE'^ + EH^) So 
(52(SoA(2)Po + PoA(2)So) 



E 

At>0 



2 A (2) 



P^B(<5)Po <5^A^- 



M>0 



-1 S^A'i^ PoB(5)P, 



Since PqH = and in view of the equality (2.9 1, we get the result. 



41 



Proof of Theorem [2^6] 

As it is shown in the demonstration of Theorem |2.5 



oo 

— 1 — '{h,---,tp+i) 

where ; +i) stands for the sum over all positive integers li, . . . , Ip^i such 

that li + . . . + Ip^i — p and li, Z^+i 7^ p. 
Let us calculate the sum 

00 

E(-i)' Er , , s^oMs)si;^^ ■ ■ • B(5)s|,'-+^\ 

p=2 

where / +1) stands for the sum over all positive integers Zi,...,Zp+i 

with li + ... + Ip^i = p such that for some 2 < k < p either the numbers 
h, . . . ,lk are positive and Ik+i = . . . = Ip+i = or Zi = . . . = Ip+i-k = and 
lp+2~k, ■ ■ • , > 0. Both variants are treated similarly. 

Denote for brevity ~ (^S^A''Q ^/fij^ . If the numbers li,...,lk are 
positive and Ij ~ for j > k, then 



P,.B(.,P. (^'A»V- 



p—k 



Ml>---,Mfc>0 '^'^ j = l 
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For fixed > 2 we have 

p>k li + ---+h=p, lj>0 ^'^ j = l 

- E E j.^^^^M^nM, 

Ml,---,Mfc>0 '^'^ lk = l 5 = 1/3=1 

l-ii,....,fik-i>Q 



k 



Since the case fc = 1 corresponds to Li((5), then the sum of (6.9 1 over k > 1 
gives us Ti. The term Tj corresponds to — ... = Ip+i-k = and 
Zp-l-2-fc, ■ ■ ■ , Ip+i > and is calculated in the same manner. 

Let us consider the residual P;]-((5) — Pj]- — T((5). As it was mentioned in the 
demonstration of Theorem 2.2 for any li, . . . , Ipj^i with li + . . . + Ip+i — p there 
exists at least one j such that Ij > and either Ij-i = or Ij^i = 0. It is easy 
to understand that such j is unique iff either h > for i < j and h = for i > j 
or li = for i < j and > for i > j. 

This means that any term 

(-l)P slj'^B{d)S^o''> . . . B{S)s'(j-+'^ 

of the residual Pi}"((5) — Pq — T{S) corresponds to the vector {li, . . . ,lp+i) 
such that for some positive kjj with i < j each of the triples 
{Ij-i, Ij, Ij+i) contains a unique zero. Denote 

m^l' ^""^ = °' and = P = 0, 

for/i_i==0 for/j_i=0 

and consider the pairs (m, m + 1) and {k,k + l). Note that these pairs are either 
disjoint orTO + l = i + l = fc = 7 — 1 with Im+i ^ h = 0- If pairs are disjoint, 
then 

||S^'^^B(,5)S^'^\.. B((5)S[,'''+^'|| 

= II S^'^^ . . S[,'"^B((5)S|,''"+'' . . . S[,''-^B(J)S[,'''+^^ . . . 8^'"+'^ II 



< 



|SoB(5)Pof (||B(5)||^„,i„) 



p-2 
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If; 



I = k, then Z„i, lm+2 > and 



(Wi)| 



s('^^B(<5)...S^""^B(<5)PoPoB(<5)S^' 



2(U) 



(/™ + 2) 



.B((5)S^'''+^^| 



.p-2 



<||SoB(5)Po|| (||B(<5)||/^^in)' 
The rest of demonstration is the same as in Theorem 12.51 

Proof of Proposition |3.1| 

If e ^ 0, then 91 ^ and 

||B((5)||//imin < 2|(5|9 + (52 99i - 2|(5|9 ^ 



for all S. This means that we can use inequalities (2.161 and (2.241 for any fixed 
6 provided that N is sufficiently big. The inequality (3.11 follows from (2.161, 
(2.171. In the same manner inequalities (2.221. (2.17) provide that 

P^ - Wi((5)|| < 64C(52. 



limsup 9 

N 



Since Wi(5) = dVl;> + 52(p„a(2)So + SoA(2)Po) and ||SoA(2)Po|l = 
||PoA(2)So|| < 99i < 92, then is proved as well. 

Proof of Proposition |3.2| 

It follows from the convergence ||HE'r||//i„,i„ ^ that 

limsup ||B(5)||/^„iin 

< 2 1 <5 1 lim sup 1 1 HE^ 1 1 /^„ii„ + (5^ lim sup 99i (5 V A < 1 /4. 

Therefore, ( |2.20| holds for \S\ < 60 and big N. Moreover, 1/(1 - 
4||B((5)||/^inin) < 1/(1-4A). Thus ([331) follows from (|2^ and (|6Jl. Applying 



( |2.24| ) instead of ( |2.20| , we get ( |3.4| . 



Proof of Proposition |3.3 

In view of Lemma 



3.1 



(a) „2N 



and /i„ 



rf{a) 2N 
mill p 



Therefore, 



^max/ max ; 



1 — \/^max/ /^max ^ ^ ^max / ^ii: 

92 = Mmax//^mi„ ~ (Til/Ti'^i) (ai/ap)^^, 
9 = 9i92 ~ VI^, 
and both assertions follow from Proposition |3.1| 
Proof of Proposition |3.4| 



1. In view of Lemma 3.2 and Remark 3.3 j/max/^lniV < 7'S' for N > No{y , uj), 
7' > 7o, and ui £ fto. This means that 

''max VMmaxA/'lnA^ 



9 



''max /^max 



iVlniV 



< 



^ ^lainxN In N/ fi 

n 
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on rio as — ^ CXI . The rest follows from Proposition |3.1[ 

2. Since Vma.x/N — Cmax with probability 1, see Lemma 3.2 and 

Mmax/Mmin = o{N~^), then 



e = 



^max Mm ax 



/^max Mmin V /^min 

and all we need is to apply Proposition |3.1[ 



a.s. 



Proof of Proposition |3.5| 

To demonstrate this assertion, we use Theorems |2.3| and |2.6| as well as the results 
of Lemma |3.2| and Appendix 2. 

Since \\B{S)\\ < 2\S\ ||HET|| + S^v^.i,^ and 



2^0 ||HET|| + J2^„ 



< 1/4, 



then the inequality (2.15 1 holds for any S G {—Sq, Sq). It follows from Proposition 



7.2 of Appendix 2 that almost surely 



limsup(iVlnlniV)"i/2 < 

N 



(6.10) 



with some positive constant ci. Therefore, ||HE"''||//i„iin— >0 with probability 1. 

As it follows from Lemma 3.2 fmax/^^ 1 with probability 1. Therefore, 
there exist ili E T with P(rii) = 1 such that for any a; G fii 



limsup ||B((5) llZ/imin < limsup ( 2S 



N 



N 



I HE J 

/^min 



Mil 



= <5V(LoA^i„). 



Thus (2.151 is proved. Consider the right-hand side 

l|SoB(,5)Po|| 

1-4||B(,5)||/Mmin 



(6.11) 



of the inequality (2.15). We have already proved that 1 — 4||B((5)||//iniin > 
1 - 4(52/(A,„i„Lo) > for \S\ < <5o, w G f^i and N > No{uj).) 

Consider the numerator of the fraction (6.11 1. It is easy to see that 

P^A^^'Po 



SoB(5)Po - S SoHE^Po + 6^J2^''' 

M>0 



In view of (6.10 1, 



limsup v/A^/lnlnATllSoHE'^Po 



N 



< limsup v/A^/lnlniV||HE'^||//i„,i„ < ci/(A„,inLo) 

N 
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for uj E ill. On the other hand, since P^Pq = 0, then it follows from Proposition 



7.3 of Appendix 2 that 



lim sup 



|PmA(2)Po| 



N 



< lim sup y/N/ In In N 

N 

= lim sup 

N 



|A(2 



A(2) - A^l)Po| 

Mmin 

-A^lll 1 



1 



A„ 



lim sup 

iLq at 



N 

|A(2)/iV-] 
VlnlnA^ 



A 



min^O 



< Ci 



with probability 1. Note that the number of positive eigenvalues of the matrix 
HH""" does not exceed 2p, thus (3.111 is proved. To prove (3.121 it is sufficient 
to use (|2.26| instead of (|2.15|). 



Proof of Proposition 4.1 



We present only the sketch of the proof. Note that the right-hand side of (4.2 1 
is 0{N'^a-'^^) if L/N ^ a € (0,1) and has the order 0{Na-'^^) in the case 

min(L, K) = const > 1. Let us study the asymptotic behavior of Vq^-*. 
Accurate calculations show that 



N 



^N+l 



a{a+l)\/a^ - 1 for -> a e (0, 1), 

for K 




Ko > 1 



and 



(a+1) 



LoWWlJ 



\\Wl„F 

for L = Lq = const > 1. This completes the proof. 



_jv 

- a 



Proof of Proposition |4.2| 

Simple calculations show that ||HE''"||//i„ 



= if X is even and that 



for even L and 



||HET||//i,„i„ = l/K for odd K. Additionally, ||SoA(2)| 
||SoA(2)|| = 1/L for odd L. 

The assertion of Theorem 2.5 holds under the condition 2\S\ |jHE''"|| + 
'^^^'max < Mmin/4. If K is cvcu, then this inequality is equivalent to 6^ < 1/4. If 
K is odd and tends to infinity, then we asymptotically get the same restriction 
52 < 1/4. 

Since the right-hand side of (3.3 1 has the form 0(||HE'^||//^,nin + HSoA'^^ ||), 
the proof is complete. 
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Proof of Proposition 4.3 



The first assertion follows from the equality ~ \/ — Pl- 

Let us prove the second assertion. Straightforward calculations show that 



|SoB(<5)|l ||SoB(5)Po|l < 



K LK 



\5\ 



L 



and 



|B(5)||/Mmin < 1^1 pk^L-^-pL/LK + 5^<h \5\IK + 5^ 



Therefore, it follows from (2.241 that for any \5\ < 1/2 

||P^(5)-P([-L(<5)|| 
(O {K--') 



O (i-) 
0(L-^+K- 



for odd K ^ oo and even L, 
for odd i — !■ oo and even K, 
2\ for odd K, L 
' and min(L, JiT) — > CXI. 



Comparing L((5) with M((5) we see that L((5) — M(5) in the case when L is 
even and K is odd. Otherwise, 



L((5) ^ M{5) 
'0{L-^) 



for odd L — > oo and even K, 

for odd K, L 

and min{L , K) oo . 



This completes the proof. 



Proof of Proposition |4.4| 

It follows from conditions of Theorem 
where 



2.6 



that P(f7 



N 



> 



\C{S) < 1/4), 



C{S) = 2 



IHE^I 

LoK 



LoK 



Since almost surely ||HE'^||/A"^0 and \\A\\/K ^\\A\\/N ^1 as N ^ oo, then 
f{nN) 1. 

Now let us turn to the operator T{S), see ( |2.26 1. First of all, P^ 



Wlq WJ^I Lq. Thus this projector does not depend on N. We get from (2.231 
and (|2.27| that 



i^T,(.)^f:(-ir(M^)'(y]^^^^ 



(6.12) 
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where 



5 



LnK 



Po + S^P^ — Pa 



HET 



Po + <5Pf^— Po 



if 



and 



A (2)^ 



(6.13) 



Let us consider the operator \'KP^'B{5)PQ/fj,, see the right-hand side of 
(6.121. Since HE'^ = {ipij^^^^^ with ipij = Sj + . . . + Ek-j-i and since 

{ej + ... + SK-j-i) = (eo + . . . + ex-i) + D{K) 
with D{K) — > in probabiHty. we obtain that 



C 



N 



HE J 



CiL^^P, 



where ^ G N(0, 1). In view of the equaHty Lq ^ P^Pq = this means that 



5 HET 

K^^^Pq 



0. 



Consider the second term of the right-hand side of (6.13 1. Since EA^^' / K = 
I and in view of the third assertion of Proposition 7.3 (see Appendix 2), we have 

A(2)-EA(2) 



^ A (2) 



Pn\=C 



Lo 



-P^VK 



K 



£( — P([*L„Po 



Therefore, 



C[^/KP^■B{5)Po|LoK) 



£( — P([vi/^„Po 



(6.14) 



Since 



PmB(<5) 



= 5P, 



HET + ERT 



, A(2) A2 , 



we see that (P^B((5)/^)'— > (5^/Lo)*Po ^i^h probabiHty 1 
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In view of Proposition 7.3 I — / ^ ^ I — (5^Po/io almost surely, 

ity : 



Therefore, with probability 1 



1-5 



2 -^0 



I^Po 

^0 



fc=i 

I Po 



1 



1 - 5Vio 



Po = P,[ 



Lo 1 - 

1 



Po 



1 - 5Vio 



and 



I -52^ 



1 



1 - S^lLo 



Po a.s. 



Collecting all these results, we see that \/NTi{5) converges in distribution to 
the random matrix 



i=0 



2 \ ^+1 



1 - <5Vio 



P([*Lo Po 



1 A2/r (l-'5V^o)Pf^*LoPo = ^Po^*LoPo. 

Since T2{5) = T^(J) we obtain that 

£ (ViVT(5)) £ ("^P^^i^ Po + Po*io Pc^) 



In view of ( |2.26| and under suppositions of Proposition |2.6[ Pj]"((5) — Pj" 
T((5) = R((5), where 



|R((5)|| <C 



|SoB(5)Pq 



= c 



|P^B(5)Po||V(ioif)' 



1-4||B(5)||^ l-A\\B{5)\\lLaK ' 

We have already proved that 4||B((5)|j//iinin '^S'^/Lq < 1 with probability 1. 
Also, SoB(5)Po — with probability 1 (see demonstration of Proposition 3.5 1 
and in view of (6.141 we obtain that 



XSoB(<5)Po = VK 



P(^B(5)P0 d^,2^± 

LoK 



Thus y7V||SoB((5)Po|| and ViV||R((5)|| ^ a.s. Since Lq is fixed, then 
VNR{S) and 

/:(N/i^(T(5)+R(5)))=^/:( — (P^*L„Po + Po*LoP,} 
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In view of the convergence P(r2Ar) — > 1, conditional distributions 

c(^VK{Ti6) + RiS))\nN) 
have the same weak limit as C (^Vk(T{S) + R((5))^ , and the proof is complete. 



Proof of Proposition |5.1| 

Note that 

R{6) -R=- (Po(^) - T>o)cL 

+ (iipoWclII^ Pw) "^"'^ ^''^ 

Since IIGlII = 1, Po((5) - Po = P(^ - P,^(<5), and UPoWclH > ||PoeL| 
AP((5), then 

||Po(<5)eL|| / llPoeill > 1 - AP((5) / ||PoeL|| , 



APffl / llPpez.!! y^AP(^) AP(<5)^-^ 



and 



11*11 < ' ^ 

< 



IIPoClII^ V||Po(<5)eL||2 
1 / 1 



< 



IPoClIP V(l-AP(5)/||PoeL||)2 
2AP(5) / AP((5) 



Thus (5.5 1 is proved. 



Proof of Lemma 15.11 

1. Denote U = \Ui : . . . : Ud\. Since Ui, . . . ,Ud are linearly independent, then 
PVX = UX 7^ for any X 7^0. Consider the vector QVX = {Q-P)VX+UX. 
Then 

||QUX|| > ||UX|| - ||Q - P|| ||UX|| 
= (1- ||Q-P||) \\VX\\ >0. 

2. Consider the linear space PV. Evidently. PV is a subspace of U. Due to 
the first assertion, the dimension of PV equals d. Therefore. PV = U. This 
completes the proof. 

Proof of Proposition |5.2| 
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Denote for short X = UTFiU, Y = UTF2U, X{S) = \J'^P^{5)FiF^{S)\J , and 
Y((5) = UTpJ^(5)F2Pff((5)U. Note that \\Y{S)\\ < Then 

f>{d) - D = l^-\S)Y{S) - X^Y 



= {X-\6)-X-')Y{6)+X-\Y{6)-Y) ^ Ji + J2. 

Since Y((5)-Y = UTAP(,5)F2P^((5)U+U'rpJ-F2AP(,5)U, then ||Y((5)-Y|| < 
2||U|p AP((5) and we obtain the inequahty 

,, :^iiui 

^^2 




In the same manner, ||X((5) 
2AP((5)/u < 1. Therefore, 

X-\5)~X-^ = 



and 

||(X-i(5)-X-i)Y(5)|| 
^ IIYfflll ||X-i(Xffl-X 
- ||X|| l-||X-i(X(5) 

The proof is complete. 



Xj| < 2||U|| AP((5) and ||X-i(X((5) 
X-i(X((5)-X) 



X)| 



< 



X 



X 



< 



i;2 1 



AP((5) 
2AP((5)/i; 



Proof of Proposition 5.3 



The first assertion follows from Remark |5.2[ L et us prove the assertions 2-4. 
Since ||H|| = ||E|| = ^/LK and in view ofltel. 



|(Po('^) 



P^ 



M(5))H(5)| 



O (yZJlO + ^K/L- 



for odd K, even L, 
and L = o{K^), 
for odd L, even 
and K = o{L^), 
for K and L both odd and 
— >■ 00. 



This means that we must study the entries of the matrix 5(M((5)H((S) + (5P;j"E). 
Calculations show that under denotation Ve = W^Ej^ and We{5) — Ej^W'^ + 
SVe, 

M{5)H{S)+dP^E 

S WEi6) 



1-52 ^ 
WEjS) 

5 /I 5_ 
1-^2 [k^L 



{We{5)+0{L-^))+5 



Ve 
L 



for odd K 
and even L, 
for odd L 
and even K, 
for X, L 
both odd. 
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Now the result becomes clear. 



Proof of Proposition |5.4| 

Let us consider the case L/N a € (0, 1) (other cases are studied in the same 
manner). It is easy to see that 

A(^V«) -J(vWh + Pf^E) +52vWe 

and 

' ' { WWlUWkW^^ \\Wk\\^ ^ \\Wl\\' 
Therefore, 

llF^l-^) -F^||_^ = X,{5vl^^)+0{N'a'^), 

where Ai(5vW) = ||5Ai(<5vW)||_. 

Consider the series go,...,gN-i corresponding to the Hankel matrix 
iSAi (i5Vq^^) . Since hankelization does not modify extreme entries aoo and 
of any n x m matrix A = {ay }"=o!j=0' ^^^Y t° ^^^^ 



i 



PlPk N-l 



II w. 

Thus the proof is complete. 



7 Appendix 2. Trajectory matrices of stationary 
processes 

Let the "noise series" E be a stationary random series with Ee„ — and De„ = 
l.(Then serves as the variance of the perturbation series SFi.) Denote E = 
El, if the corresponding random trajectory L x K matrix. 

In this section we present an upper bound for HE^.k!] under supposition 
that e„ is a linear stationary process. More precisely, we suppose that 

oo 

j = -(X> 

where ej is a sequence of independent random variables with Eej = 0, De^ = 1 
and sup„E|e„p < oo. In addition we assume that \cj \ < oo. Since De„ = 1, 
then J2j c| = 1. 
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Note that this model of a stationary process is widely used in statistics (for 
example, see [26] ) . In particular, if we put 



E 



ap 



(7.2) 



with IpI < 1 and a = \/l — , then we get the first-order autoregressional 
process e„ = pe„_i + a£„ with the covariance function Re{n) — p". Thus the 
important "red noise" example is the particular case of (7.1). 



Section [7.1 1 is devoted to general upper bounds for norms of trajectory matri- 

random variables 



ces of linear processes. In Section 7.2 the special case of i.i.d 
e-n = En is considered in a bit different manner. 



7.1 Linear processes 

Let P) be a certain probability space and consider a sequence e„ {n = 

0,±1,±2, . . .) of random independent variables defined on J^, P). Assume 
that Ee„ = 0, D£„ = 1, and sup„ E|£„p < oo. Denote for n = 0, 1, 2, . . . random 



variables e„ by formula (7.1 1 under condition J2j c| = 1- Lastly, let N oo, 
L = L{N), K = N + l-L and denote El,k = {ei+fe-2}i<j<i, i<j.<x the LxK 
trajectory matrix of the series E = (ep, . . . , e„, . . .). 

Proposition 7.1. If S = < oo. then there exists fi'^^^ G with 

P(17(0)) = 1 such that for any uj G fl'^^') 

limsup - — '"'^^ •^11 < 'jqS, (7.3) 
N VNlnN ~ 

where 70 > is an absolute constant. 

Proof. Our proof is essentially based on results and ideas published in |22| . 
Since we need only an upper bound for ||Ei ||, we do not use final results of 
|22| . dedicated to the precise rate of growth for norms of square Hankel (and 
Toeplitz) matrices with i.i.d. entries. Still the line of our considerations is 
borrowed from [22 . 

We start with mentioning that 

||ELAic|| < ||Ei_j<-|| < llEivifll, (7.4) 

where we use denotation E^v/ instead of 'Em,m- (As usual, L A K = min(L, K) 
and L\/ K = max(i, K).) Indeed, this follows from the remark that E^ak can 
be considered as a submatrix of Elk and that El.k is a submatrix of Eivif- 
The second step is to consider the infinite Laurent matrix 



L„(E) - {eK-fe|lb-/c|<n-i} - fc^o. 



±1,±2,, 
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(Note that L„ depends on ej only for < j < n — 1.) Then 

ji-i 

eo + ^ Cj cos(27rtj) . (7.5) 



L,J = max 

' " 0<t<l 



This fact is a variant of basic relationship of Toeplitz matrices to multiplicative 
operators in L^[0,1] (see [27], the short outline of the proof can be found in 
|21j). Note that here L„ is considered as the operator on ^^(Z), where £'^{Z) is 
treated as a set of two-side infinite complex sequences b = {bj}j^z such that 
l|b|P = E,al^.P<oo. 

Laurent matrices L„ are useful for our goals since 

||E„|| < ||L2„_i(E)||. (7.6) 



To prove (7.61, consider the Lx_fC matrix R^^a' = {^j-i^+K-i} i^^^j^ i<k<K' 
This matrix is obtained by reversing of columns of Hankel matrix El^k, there- 
fore singular values and the corresponding left singular vectors of matrices El,k 
and Rl.k coincide. In particular, HRl^^H = ||EL_/f||. 

If we take L = K = n, then we see that R„^„ is a submatrix of the Laurent 
matrix iv2n-i- Indeed, the matrix i?„_„ is placed on the intersection of the set 
/ = { — (n — 1), — (n— 2), . . . , —1, 0} of rows and the same set J — I of columns 
of the Laurent matrix L2n-i- Thus ||E„|| = ||R„.„|| < ||L2„-i||. Note that this 
procedure was discussed in p5| . 

Suppose now that E = (eo,£i, . . . ,e„, . . .) is the sequence of independent 
random variables with zero means and unit variances. Then it follows from |221 
theor. 4] that there exists an absolute constant 71 > such that 

E||L„(E)|| < jiVnlnn. (7.7) 

If we assume additionally that sup„E|e„p < 00, then 

||L„(E)|| E||L„(E)|| 
hmsup —^^^ < hmsup — -^^^^ (7.8) 
n \/nlnn n V n In n 

with probability 1. This fact is proved in the demonstration of |22l theor. 5] 
under more weak moment conditions on e„ . We have simplified these conditions 
to make them more transparent. 



Collecting inequalities (7.4 1, (7.6 1, (7.7), and (7.8 1 we come to inequality 



dcf 



(7.3 1 for the particular Indeed, let n = max{L, K). Then 



[iV/2] < n < iV and 



I|El,a:|| ^ ,. ||E„|| ||L2„_i|| 

lim sup ; < um sup ; < um sup : 

N y^NlnN n VN InN n VN In TV 

IIWill V(2«-l)ln(2ri-l) , ^ 

< Imi sup , — , 7 g) 

~ N ^(2n- l)ln(2n- 1) y/NhiN ^ ' 

/ r V^||L2n-l|| ^ /t; 

< hmsup , = < v27i = 7o, 

" 2„-i ^(2n- l)ln(2n- 1) " 



54 



where 71 is described in (7.7 1 and the last inequahty holds with probability 1. 



The last step is to return to the linear process under consideration. Since 



(m) 



where E 



(m) 
L.K 



E 



L.K\ 



< 



El 



<j<L, l<k<K 



, then 



E 



(m) I 

l,k\ 



Of course, inequality ( |7.9[ ) holds for operators E^"^. Taking into account 
the fact that 70 is an absolute constant, we come to (7.3 1. □ 



Remark 7.1. 1. It follows from Proposition 

IIEl,kHI 



7.1 



that for w e 



max 

N 



with 705 < c{uj) < CO. 
Nq = No{u!, a') such that 



VNlnN 

On the other hand, for any a' > there exists 



IE 



max 



L,K 



Mil 



< a' S. 



2. Of course, S = 1 for the white-noise variant of e„. It is easy to check that 
S = 1/(1 + |p|)/(l — \p\) for the autoregression process ( 7.2 1 with a — \J\ — p^. 



This means that the upper bound in the right-hand side of (7.3 1 tends to infinity 
as \p\ 



1. 



3. Inequality (7.3 1 holds for the arbitrary behavior of L = L{N). Yet in some 
cases this inequality can be refined. For example, if rt = max(i, K)/N — > /3 < 1 
as — ^ 00, then the right-hand side of (7.3 1 can be multiplied by Since 
(3 > 1/2, the advantage is not very big. 

4. The case when L = Lq — const is much more simple. Of course, 
is equal to largest eigenvalue of the Lq x Lq matrix E^p ^'Ej^ ^. 

Entries of the matrix Eig^^fEj^ k/^ have the form 



E 



0^ 



1 

m=0 



i,j = 0, . . . ,Lo - 1- 



(7.10) 



If the series E is strongly stationary and ergodic (see 
definitions), then the series E^^ = (eiCj, ei+iej+i, . 

strongly stationary and ergodic and therefore (j))^'^ 
Re{i — j) = EiCiCj as K ^ 00. 

This means that HE^^ /^Ej k\\/^ tends a.s. to the maximal eigenvalue 



ch. 5] for precise 

■ T ^i+mGj+m, ■ ■ ■) is also 

converges almost surely to 



a<t,j<Lo~r 



Therefore, ||ELujf||vA^ — > V A^ax with probability 1. In particular, if e„ are 
i.i.d. random variables with Ee„ = and De„ — 1, then E^^jf Ej^ k/^ 
verge a.s. to the identity matrix I^^ and HE^o./fH 1 with probability 1. 
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5. These examples show that normahzing factor l/y/N\nN in the lefthand side 
of (7.3 1 is not uniformly precise. On the other hand, it is proved in |22l corr. 



3 and sect. 3.4] that in the case of i.i.d. random variables with zero mean and 
unit variances, 



Imi sup < oo 

n y/ nlnn 



with probability 1. Applying inequality (7.4 1, we see that the same result is 



valid for the sequence of rectangular matrices 'El.k under supposition that 
mm{L, K)/N ^ a>0. 



7.2 The case of i.i.d. random variables 

Consider a probability space (O, J^, P) and a sequence of i.i.d. random variables 
£0, £i, . . . defined on this space. Assume that Ee^ = 0, De^ = 1, and supj Eef < oo. 

Let us present several auxiliary assertions related to the series E. For con- 
venience we collect these assertions in the following propositions. 

Denote 



= ^ 7fe cos(27rwfcn + (pk) 



k=l 



with 7fc and coj ^ ujk for j ^ k. 
Proposition 7.2. For any m, Z > 



lim sup 



En — l I 
j^O Xm+j£l+j I 

V n In In n 



\ fc=i 



(7.11) 



with probability 1. 

Proof. For fixed m and I set = x^+jSi+j- It can be easily checked that 

?i— 1 p 



q2 ciof 



^„_i) = ^ xl^.^ - n ^ 7,2/2 + 0(1) 



k=l 



Since sup.- E|^j|'^ < oo, then it follows from [30, ch. 17] that almost surely 



lim sup 



En— 1 I 

-B„\/2nln Inn 



Thus is proved. 



Denote 



□ 



in the manner of (7.101 changing e„ in this formula for e„. 
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Proposition 7.3. Suppose that Ee^ < cjo. 

1. If j = j and n ^ oo, then E<j)[f = 1 and nD(p[f = =^ Ee^ - 1. Besides, 
— >■ 1 almost surely. If i ^ j and n — > oo, then E(j>[^^ — and tiDi^I"'' — > 1. 



Besides, 0^"'' — almost surely. 



2. Denote p'^^\i, j\ s,t) the correlation coefficient between cjy^ and . If 
|j — i\ = |t — s|; then p'^'^\i, j; s,t) — > 1 as n — cx). If |j — i| 7^ |t — s\, then 
p(")(i,j;5,t)=0. 

3. Let M be fixed and n ^ cx). Denote {'/'ij^ll^j^o- Then ^ I with 
probability 1 and the limit distribution of the sequence of random matrices 
y/n{^n ~ I) coincides with the distribution of the random M x M matrix '^m = 

''^ch that 

a) ip.j = V'st for \i - j\ = |s - t\; 

b) random variables -011, ipi2, ■ ■ ■ , ^im are independent; 

c) Vii G N(0, A2) and 0^ G N(0, 1) for i > 1. 

4. If i = j, then 



lim sup \/n - — " = v Ee^ 

n V In In n 

with probability 1. If i 7^ j, then almost surely 

r- 

lim sup V n = 1 . 

n V In In n 

Proof. 1. The assertion follows from the fact that random variables eQ,ef, . . . 
are i.i.d. with Ee^ = 1 and De^ = A^. The a.s. convergences of <p[^^ are already 
discussed in Remark |7.1[ Other assertions are checked straightforwardly. 
2. Let i < j and i < s < t. (Other variants are treated in the same manner.) If 
j~i~t — s~h, then E^^"-* — E^^"-*, D^^"'' = D0^"\ and under denotation 



for n big enough 



n ^ 

m=0 



_j^n — 1 — 1 



— ^ ^ ^i+m^i+/i+m — ^ ^ ^ i-\-ra£ i-\-h-\-m ~^ 

■> n ^-^ n ^-^ 

m— m—s — i 

n— 1— s+i 



n ^-^ 

^ n— 1 ^ n— 1 

— ^s+p£t+p + ^n £s+p£t+p^4'it^ +Bn 

p—Q p—n—s+i 
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with EBl = 0(l/n2). Therefore, E0,^;Vi"^ = + E(0l; ^B„) 



and (j, j; s, ^ 1. If j-i^t-s, then E(/)|j%(/)^"^ = 0, Eei+rnSj+m^s+pSt+p = 
for any nonnegative m,p, and p^'^\i, j; s,t) = 0. 

3. The convergence $„ — > I immediately follows from the first assertion of the 
proposition. 

If i — \t — s\, then 

p 

where C„ — > as ri — > oo. This means that all we need is to prove that 
coordinates of the random vector 

V^(</.[,^) - E</)(^', . . . , - E4,^^l,_,) (7.12) 

are asymptotically independent and asymptotically normal with proper vari- 
ances. 

Denote for < j < Af - 1 



n-j-l 



(n) _ 1 



m=0 

Then Erj"^ = E0[,"j and the difference ^/^{4>^^} - rj"^) tends to zero in prob- 
ability as ri — > oo. This means that the asymptotic distribution of the random 
vector 

V^(r(") - Er("\ . . . , r^l, - Er^^,) (7.13) 

(provided that this distribution exists) coincides with the asymptotic distribu- 
tion of the vector (7.121. 

(n)^ 

Note that is the usual unbiased estimate of the covariance function 
of the stationary series E. Due to |26| th. 8.4.2], distribution of the vector 
(7.131 weakly tends to Af-dimensional Gaussian distribution with zero mean 

I M-l 



and covariance matrix W = {y^ki}^ /-o' ^^^^^^ 



Wki^- I cos(fcA) cos(a) d\ + {Ee^ - 3)So{k)So{l) 



TT 



and (5o(j) is either 1 or depending on the conditions j = and j ^ 0. Since 

{0 for fc 7^ I, 
TT for fc = Z 7^ 0, 
27r for fc = / = 0, 
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we see that wqo = Ee"* — 1, Wkk = 1 for k 0, and Wki = for k ^ I. 
4. The first equahty is the particular case of the standard Law of Iterated Log- 
arithms (LIL) for i.i.d. random variables. To demonstrate the second equality 
it is sufficient to put i = 1. Then under denotation N = [n/j\ 

j N-l j 
fe=l 1=0 k=l 

Since sl^^ is the sum of i.i.d. variables we can apply the mentioned LIL to 
this sum and asymptotically change N for n/j. As Dr„ is bounded above, this 
completes the proof. □ 
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